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Executive Summary 


The Florida Master Teacher Initiative (FMTI) is a collaborative professional development effort 
of Miami-Dade County Public Schools, the University of Florida (UF), and the W.K. Kellogg 
Foundation aimed at improving early learning instruction for high-need preschool through 
grade 3 students in Miami-Dade County Public Schools. Funding from the U.S. Department of 
Education’s Investing in Innovation (i3) program supported the development, implementation, 
and evaluation of FMTI. SRI International (SRI) conducted the independent evaluation of FMTI. 

Program Overview of the Florida Master Teacher Initiative 

FMTI aims to enhance early learning instruction for high-need students through four program 
components: 

• A job-embedded graduate degree program with an early childhood specialization—the 
Early Childhood Teacher Leadership for School Improvement (ECTLSI) Program— 
offered through UF. 

• A Teacher Fellows program, through which a subset of teachers throughout the school 
engage in yearlong inquiry projects to examine new instructional approaches together 
with peers and present their projects at an annual districtwide Learning Showcase. 

• A Principal Fellows program, which provides opportunities for principals to interact with 
other principals in discussions and activities that help build their shared leadership skills 
and enhance their ability to effect change within their schools. 

• Summer leadership institutes, which support shared school leadership through facilitation 
of joint analysis of school surveys and student assessment data to support data-driven 
decision making and development of school action plans. 

The FMTI team also introduced three program enhancements in its second and third years of 
implementation, including: (1) an Assistant Principal Professional Learning Community 
(APPLC), a program similar to the Principal Fellows program but for assistant principals: (2) a 
post-baccalaureate program, a non-degree-bearing, four-course version of the ECTLSI program 
available to a wider range of teachers than those who qualify for the ECTLSI graduate program, 
and (3) a Transition to Kindergarten Professional Learning Communities to support inquiries 
about aligning standards, social-emotional development, and barriers to smooth transitions 
between preschool and kindergarten. 

SRI International conducted an independent evaluation of the impact of FMTI on school culture 
and teacher practices as measured by teacher surveys and teacher observations, and on student 
reading and math achievement as measured by standardized tests. The evaluation used both a 
cluster assignment random control trial (RCT) design, in which 40 Miami-Dade County public 
elementary schools were randomly assigned to the FMTI program or a status-quo control 
condition to assess schoolwide impacts, and an embedded quasi-experimental design (QED) 
using propensity score matching and difference-in-differences approaches to examine impacts on 
teachers who participated in the ECTLSI graduate program. The evaluation also included a 
formative evaluation to leam about the program’s implementation, accomplishments, challenges, 
and strengths to support program refinement, replication, and the sharing of lessons learned with 
the education field. The evaluation also included a fidelity implementation study that examined 
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whether FMTI was implemented as intended to provide formative feedback and help with the 
interpretation of the impact study results. Below is a summary of the key evaluation findings. 

Key Findings 

Implementation Outcomes 

• FMTI partners were successful with offering the planned programs for principals and 
teachers. All of the courses for the ECTLSI graduate degree program were developed, 
implemented, and well received. The Teacher Fellows program was successfully 
implemented in all the treatment schools. The UF team held statewide institutes and local 
meetings for principals to learn about and observe new leadership practices and develop a 
professional network and learning community with other principals of FMTI schools. 

• Recruitment and participation levels in the ECTLSI graduate degree program and 
Principal Fellows program were a challenge and resulted in the program not achieving 
the level of implementation fidelity needed to accurately assess the model’s impact. 
Further, requirements related to schools being designated high priority schools by the 
Educational Transformation Office made implementing some of the instructional 
practices learned through the graduate program a challenge. 

• FMTI partners responded to these challenges by incorporating new program 
enhancements and working on the scheduling of program activities to avoid conflicts 
with other district activities. 

Confirmatory Findings on Schoolwide Outcomes 

• Teacher outcomes: There were no statistically significant differences between teachers 
in treatment schools and teachers in comparison schools on the majority of outcomes 
measured through the teacher survey. However we did find two significant results: 

o Teachers in treatment schools were more likely to engage in governance activities 
than their counterparts in control schools (82% vs.76%) 

o Teachers in treatment schools performed slightly worse than their counterparts in 
control schools on reported use of differentiated instruction (-.08 points on a 1-5 
scale) 

• Student achievement: There were no significant differences between students in 
treatment and control schools on math or reading achievement. 

Exploratory Findings on Schoolwide Outcomes 

• Teacher outcomes in medium or high fidelity treatment schools: There were no 
statistically significant differences between teachers in medium/high fidelity treatment 
compared to teachers in comparison schools in the same regions and voting districts on 
the majority of survey outcomes. However, there were four positive and statistically 
significant results: 

o Teachers in medium or high fidelity treatment schools reported slightly higher 
perceptions of trusting relationships between teachers within their schools than 
teachers in control schools (.22 points on a 1-4 scale) 


o Teachers in medium or high fidelity treatment schools reported slightly higher 
levels of involvement in leadership roles than teachers in control schools (.36 
points on a 1-5 scale). 

o Teachers in medium or high fidelity treatment schools reported reaching out to a 
slightly higher proportion of families than teachers in control schools (.34 points 
on a 1-6 scale) 

o Teachers in medium or high fidelity treatment schools reported using a greater 
variety of assessments than teachers in control schools (.23 points on a 1-4 scale) 

Student achievement for those who remained in the same school for the full 
intervention period: There were no statistically significant differences for reading and 
math achievement for Pre-K-2nd grade students who were still enrolled in the same 
school in 2013-14 compared to similar students at control schools. 

Student achievement in medium or high fidelity treatment schools: There were no 
statistically significant differences for reading and math achievement for Pre-K-2nd 
grade students who were in medium or high fidelity treatment schools compared to 
similar students at control schools who were still enrolled in the same school in 2013-14. 

Confirmatory Findings on Graduate Degree Teacher Outcomes 

ECTLSI instructional practices: The evaluation found one positive difference in 
teacher quality for ECTLSI teachers compared to matched teachers based on the 
Classroom Assessment Scoring System (CLASS): 

o On average, observed ECTLSI teachers outperformed their matched comparison 
teachers by 1.7 points on the instructional support domain of the CLASS. 

o The evaluation found no statistically significant differences between observed 
ECTLSI teachers and their matched comparison teachers on the emotional support 
or the classroom organization domains of the CLASS. 

ECTLSI teacher outcomes: The evaluation found five positive and significant results 
for ECTLSI teachers compared to teachers in comparison schools on the teacher survey: 

o On average, ECTLSI teachers had slightly higher levels of involvement in 
leadership roles than similar teachers in control schools (.44 on a 1-5 scale). 

o On average, ECTLSI teachers performed slightly better than teachers in control 
schools on self-reported early childhood knowledge (.38 points on 1-4 scale) 

o On average, ECTLSI teachers performed slightly better than teachers in control 
schools on self-reported general instructional knowledge (.27 points on 1-4 scale) 

o On average ECTLSI teachers are more likely to engage in governance activities 
than teachers in control schools (94% vs. 76%) 

o On average ECTLSI teachers are more likely to engage in outreach activities than 
teachers in control schools (44% vs. 22%) 

ECTLSI teacher student achievement: The evaluation compared differences in student 
reading and math achievement between ECTLSI and other teachers in treatment schools 
with the difference between control school teachers matched to ECTLSI teachers and 
nonmatched control school teachers. The difference-in-differences estimate did not find a 
statistically significant difference between them on student math or reading achievement. 


While the FMTI evaluation was not sufficiently robust to definitively determine the effectiveness 
of the program, small pockets of positive, significant findings suggest that FMTI may have 
potential to change instructional practices, if not yet student achievement. 



1. Introduction 


The Florida Master Teacher Initiative (FMTI) is a collaborative professional development effort 
of Miami-Dade County Public Schools, the University of Florida (UF), and the W.K. Kellogg 
Foundation aimed at improving early learning instruction for high-need preschool through grade 
3 students in Miami. Funding from the U.S. Department of Education’s Investing in Innovation 
(i3) program supported the development, implementation, and evaluation of FMTI. SRI 
International (SRI) conducted the independent evaluation of FMTI. 

The evaluation had three primary goals: (1) to provide formative feedback to support program 
refinement and replication; (2) to assess program implementation; and (3) to measure the impact 
of the program on teachers and students. This final evaluation report summarizes the formative 
findings, describes the extent to which FMTI was implemented with fidelity, and presents 
findings on the impact FMTI on teachers and students. Previous reports described the formative 
findings and implementation of the program in more depth. 1 

In the remainder of this chapter, we provide an overview of FMTI and describe the evaluation 
design and questions. 

Program Overview of the Florida Master Teacher Initiative 

FMTI aims to enhance early learning instruction for high-need students through four main 
components: 

• A job-embedded graduate degree program with an early childhood specialization—the 
Early Childhood Teacher Leadership for School Improvement (ECTLSI) Program— 
offered through UF. The graduate program combines online instruction, face-to-face 
pedagogy, and a professor-in-residence who works in schools with teachers and 
principals. The program also provides the graduate students with training on facilitating 
professional learning communities, guiding teacher inquiry, and using formal protocols to 
guide meetings to enable the graduate program teachers to take on school-based 
leadership opportunities and share program content and practices with colleagues. 

• A Teacher Fellows program, through which a subset of teachers throughout the school 
engage in yearlong inquiry projects to examine new instructional approaches together 
with peers. This program culminates in a districtwide Learning Showcase in which 
teachers—as well as principals, assistant principals, and other school staff who also 
engaged in inquiry—present their projects. 

• A Principal Fellows program, which builds leadership skills and provides an opportunity 
to interact with principals from other Florida school districts to support teacher leadership 
and instructional improvements. The Principal Fellows program supports principals’ 
efforts to adopt a facilitative leadership approach and enhance their ability to effect 
change within their schools. 


Golan, S., Wechsler, M., Park, C., and Chen, W. (2012). Evaluation of the Florida Master Teacher Initiative: 
Formative Report. Menlo Park, CA: SRI International. 

Golan, S., Warner, M. Wechsler, M., Park, C., and Campbell, A. (2013). Evaluation of the Florida Master 
Teacher Initiative: Second Formative Report. Menlo Park, CA: SRI International. 
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• Summer leadership institutes that develop shared leadership at schools through 

facilitation of joint analysis of school surveys and student assessment data to support 
data-driven decision making and development of school action plans at summer 
institutes. 

The FMTI team also introduced program enhancements following Year 1. In the 2012-13 school 
year, the FMTI team introduced an Assistant Principal Professional Learning Community, a 
program similar to the Principal Fellows program but for assistant principals. In the 2013-14 
school year, UF faculty launched a new post-baccalaureate program, a non-degree-bearing, four- 
course version of the ECTLSI program available to a wider range of teachers than those who 
qualify for the ECTLSI graduate program. Finally, the FMTI team launched the Transition to 
Kindergarten Professional Learning Communities to support inquiries about aligning standards, 
social-emotional development, and barriers to smooth transitions between preschool and 
kindergarten. 

Logic Model 

The logic model below presents how FMTI program components are expected to lead to 
increased student achievement, stronger emotional and social foundations for student learning, 
and greater student engagement. SRI developed the logic model in collaboration with the FTMI 
development and implementation team at UF. According to the logic model, FMTI activities are 
hypothesized to support the development of a professional learning community among school 
staff and the professionalism and effectiveness of teachers. The job-embedded graduate program 
aims to develop a cadre of teachers with deeper knowledge of the early learning foundations 
promoted by FMTI, as well as strong research and leadership skills at each school. This cadre of 
job-embedded graduate program teachers is then able to facilitate a Teacher Fellows program 
that supports involvement of all teachers at a school in inquiry research projects to improve their 
practice and share in learning about their early learning knowledge. By running the Teacher 
Fellows program and other professional development efforts at their schools, the job-embedded 
graduate program teachers build their facilitation and leadership skills. Principals also learn how 
to support the emerging teacher leaders and researchers through their own professional learning 
community, the Principal Fellows program. Finally, teachers and administrators come together in 
teams as Summer Leadership institutes to develop school plans before each school year begins to 
improve school climate, instruction, and student achievement. 

The main program components, together with additional program activities, are meant to produce 
teachers who excel as (1) master teachers who use effective teaching practices, including those 
focused on early childhood, (2) teacher leaders in their schools and communities who promote 
instructional improvement and high quality education from prekindergarten through third grade, 
and (3) teacher researchers who, through participation in a guided inquiry, research instructional 
topics that are relevant and important to improving their practice. 
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Exhibit 1-1. Logic Model 


Project Supports 

Early Childhood Education Graduate 
Degree Program 

Job-embedded graduate coursework 
situated within teachers' contexts and 
that meets NAEYC guidelines 
Online and in-person implementation 
support from professors-tn-res*dence 
Demonstration of Inquiry and reflection 
through completion of a portfolio of 
projects 

Training to facilitate professional 
learning communities 
Facilitation of teacher fellow program 
or other PD 

Cohort of school-based, preK-3 ,d grade 
teachers 

Teacher Fellows Program 

Year-long Inquiry project 
6 onsite facilitated learning community 
meetings to support inquiry 
Cross-school presentation of projects 
at Inquiry showcase 

Principal Fellows Program 

Professional development meetings on 
leadership skills (4 a year) 

Interactions with principals across the 
state at an annual statewide Institute 

Summer Leadership Institutes 

Leadership institutes for administrators 
and teacher leaders to review data and 
develop school plans 


Florida Master Teacher Initiative 


Professional Learning Community 

Distributed leadership 
Dedicated time for structured 
collaboration within and across grades 
Data-based decision-making about 
teaching and learning using multiple data 
sources 

Trusting relationships between teachers 

Administrator and infrastructure support 
for school-wide implementation 
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Teacher Professionalism 

Master teacher 

• Uses effective classroom practices 

• Has enhanced knowledge of early childhood 
education 

Teacher leader 

• Facilitates professional learning 
communities 

• Advocates for children 

• Reaches out to the early childhood 
community 

• Has knowledge of and cross-sector 
collaboration around early childhood issues 

• Supports transitions across earty childhood 
years 

Teacher researcher 

Engages in inquiry to improve teaching and 
learning 




Improved Classroom 
Practices 

Instructional practices that are: 

• Research-based 

• Differentiated 

• Emphasizing higher-order thinking 
skills 

• Learner-centered 

• Culturally responsive 

• Developmental appropriate 

■ Based on data and informed by an 
array of assessment methods 

• Supporting social-emotional 
development 

Positive classroom climate 
Family-school partnerships 
School-early learning partnerships 


Learner Outcomes 

Increased academic achievement 

Strong social and emotional 
foundations for learning 

Increased student engagement 


State and Local Policies 


3 
































With improved skills in teaching, research, and leadership, it is hypothesized that teachers will 
be able to establish classrooms in which instruction is more research-based, differentiated, 
focused on higher-order thinking skills, learner-centered, culturally responsive, developmentally 
appropriate, guided by data from meaningful assessments, and supportive of social-emotional 
development. Further, teachers in these schools should be more able to establish classrooms with 
positive classroom climates, establish stronger partnerships with parents and the community, and 
more effectively engage with the early education community. It is hypothesized that these 
improvements in classroom instruction will contribute to improvements in student achievement, 
children’s emotional and social development, and student engagement. 

Evaluation Design 

A goal of the i3 program is to provide rigorous research on the effectiveness of educational 
programs. SRI International conducted an independent evaluation of the impact of FMTI on 
school culture and teacher practices, measured by teacher surveys and teacher observations, and 
on student reading and math achievement as measured by standardized tests. The evaluation used 
both a cluster random assignment design, in which 40 Miami-Dade County public elementary 
schools were randomly assigned to the FMTI program or a status-quo control condition to assess 
schoolwide impacts, and an embedded quasi-experimental design using propensity score 
matching and difference-in-differences approaches to examine impacts on teachers who 
participated in the ECTLSI graduate program. Thus, the evaluation includes both a schoolwide 
random control trial design and a quasi-experimental design with a matched comparison group. 

The evaluation also included a formative evaluation that provided data on implementation 
challenges to support midcourse corrections, assess progress, document program functioning, 
and support replication. Finally, the evaluation also included a fidelity implementation study that 
examined the extent to which the program was implemented as intended. 

This section provides a brief description of the evaluation questions and methods used as part of 
the formative, implementation fidelity, and impact components of the evaluation. Appendix A 
contains a more detailed description of the methods used for sample selection, data collection 
methods and measures, and analysis. 

Formative Evaluation 

In Years 1 and 2 of FMTI’s implementation, SRI conducted a formative evaluation to learn about 
the program’s implementation, accomplishments, challenges, and strengths to support program 
refinement, replication, and the sharing of lessons learned with the field of education. The 
formative evaluation examined all four FMTI program components. 

For the formative evaluation, researchers collected data through interviews with district and 
school level staff and document reviews. In the spring of 2012 and again in the spring of 2013, 
SRI visited six schools and interviewed principals and a sample of teachers participating in the 
ECTLSI graduate program and Teacher Fellows programs. SRI also reviewed documents such as 
curricula, program newsletters, and Learning Showcase program booklets to gather information 
about program activities. Additional information about the methods used is described in 
Appendix A. 
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Implementation Fidelity 

Understanding the extent to which a program is implemented as intended is key to supporting 
program replication and interpreting the results of an impact study. To measure implementation 
fidelity, SRI worked with UF, the program developer, to identify the types and intensity of 
activities in each of the four program components believed necessary to bring about the desired 
changes and outcomes specified in the program logic model. These assumptions were used to 
develop fidelity measures detailed in Appendix B. SRI collected participation data for the 
Teacher Fellows program, Principal Fellows program, and Summer Leadership Institutes from 
the district, and participation data for the ECTLSI graduate program (including course 
completion, grades, participation in coaching training, enrollment status) from the University of 
Florida. In addition, SRI collected supplemental data from ECTLSI teachers through program 
surveys that asked about their involvement in leadership of professional development sessions 
within their schools and their engagement in inquiry projects. A summary of the fidelity model 
and the fidelity results are presented n Appendix B. 

Impact Evaluation 

The impact evaluation had two primary goals: (1) to assess the school-level impact of FMTI on 
teachers and students; and (2) to assess the impact of FMTI on teachers enrolled in the job- 
embedded early childhood graduate degree program and those teachers’ students. These goals 
generated the following confirmatory research questions: 

1. What is the school-level impact of the Florida Master Teacher Initiative on pre¬ 
kindergarten to 5 th grade teachers’ reports about school culture (i.e., collaboration within 
the school; reflection on instruction with other teachers; promotion of active learning) 
and their instructional practices (i.e., use if inquiry in instruction, use of collaboration in 
instruction, differentiated instruction, higher-order thinking skills, learner-centered 
instruction, culturally-responsive instruction, developmentally-appropriate instruction, 
and assessment-informed instruction) three years after the program has been in place 
compared to teachers in control schools? 

2. What is the impact of the Florida Master Teacher Initiative on reading and math 
achievement after three years of program implementation, for all students who were in 
grades Pre-K through 2 in the treatment schools at the time of random assignment of 
schools to treatment conditions, compared to students who were in grades Pre-K through 
2 in a control school at the time of random assignment? 

3. After the program has been in place for three years, what is the added impact on reading 
and math achievement for students in grades 1 through 3 with one year of exposure to 
teachers who participate in the job-embedded graduate program? 

4. What is the impact of the Florida Master Teacher Initiative on classroom instruction for 
teachers who have completed or nearly completed the job-embedded graduate program 
compared to matched comparison teachers, adjusting for their baseline classroom 
instructions? 

To assess the school-level impact of FMTI on teachers and students, SRI used a cluster random 
assignment control (RCT) design. In spring of 2011, SRI randomly assigned 40 schools in 
Miami-Dade County Public Schools to participate in FMTI for three years or to the status-quo 
condition. To be eligible for FMTI, schools had to be Title I schools, have a pre-kindergarten 
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program, have at least four teachers interested in the graduate program, and have no previous 
FMTI programs or other key teacher development programs. The 20 schools assigned to the 
FMTI condition began participating in FMTI in summer of 2011, and continued working with 
FMTI through the 2013-2014 school year. 

We evaluated the effects of teacher participation in the ECTLSI graduate program through an 
embedded quasi-experimental design using propensity score matching and difference-in¬ 
differences approaches. Researchers identified a matched comparison sample of teachers in 
control schools who were similar in teaching background, instructional practices, and interest in 
the ECTLSI graduate program prior to random assignment. We applied propensity score 
matching to identify comparison teachers (see Appendix A for a detailed description of the 
comparison teachers). UF recruited two cohorts of teachers for the embedded graduate degree 
program; Cohort 1 began the program in fall 2011 and Cohort 2 began the program in fall 2012. 
SRI pooled the two cohorts of ECTLSI teachers for analysis. 

In addition to confirmatory research questions, we generated exploratory research questions 
aimed at understanding the impact of FMTI under more ideal conditions. Specifically, we 
examined the impact of FMTI among schools with medium or high level of program 
implementation and on students who remained in schools for all three program years. Our 
exploratory questions were: 

1. What is the school-level impact of FMTI on pre-kindergarten to 5th grade teachers’ 
reports about school culture and their instructional practices three years after the program 
has been in place for teachers in schools that meet the implementation threshold 
compared to teachers in control schools? 

2. What is the impact of FMTI on reading and math achievement after three years of 
program implementation, for all students who were in grades Pre-K through 2 in 
treatment schools at the time of random assignment of schools and remained in the same 
schools all three years of the program, compared to students who were in grades Pre-K 
through 2 in a control school at the time of random assignment and remained in the same 
schools in the third year of the program? 

3. What is the impact of FMTI on reading and math achievement after three years of 
program implementation, for all students who were in grades Pre-K through 2 at the time 
of random assignment of schools in treatment schools that meet implementation threshold 
and remained in the same schools all three years of the program, compared to students 
who were in grades Pre-K through 2 at the time of random assignment in control schools 
that are comparable to treatment schools that meet the implementation threshold and 
remained in the same schools in the third year of the program? 
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Measuring Teacher and Student Outcomes 

SRI administered schoolwide surveys, conducted classroom observations of ECTLSI graduate 
program teachers and a matched comparison group, and gathered student achievement data for 
analysis at multiple levels. Appendix A contains a more detailed description of the methods used 
to collect and analyze teacher and student outcomes. 

Teacher survey. SRI collected survey data from instructional staff both at baseline and 
following the three-year implementation FMTI. Researchers administered surveys in all 20 
schools participating in the program and 20 control schools. The survey captured teachers’ self- 
reports of instructional practices and school culture from the prior school year (the 2010-11 
school year at baseline and 2013-14 school year at follow-up). 

Exhibit 1-2 presents the response rate for 2011 and 2013. SRI surveyed a total of 1,397 teachers 
of prekindergarten through 5 th grade students in 2011 and 1,335 teachers in 2014. Non-classroom 
teachers (e.g. media specialists, instructional coaches, special area teachers) and administrators 
are excluded from the survey sample. All 40 schools responded to the survey. 


Exhibit 1-2. Teacher Survey Sample 



2011 

T reatment 

2011 

Control 

2014 

T reatment 

2014 

Control 

Total survey eligible respondents 

685 

712 

643 

692 

Total number of respondents 

596 

631 

565 

634 

Responses rate 

87.0% 

88.2% 

87.9% 

91.6% 


Teacher survey data provided information on teachers’ professional background, the professional 
learning culture of schools, teachers’ involvement in school leadership, teacher engagement in 
trying and evaluating new instructional practices, and the use of various classroom practices. 
Sixteen outcome measures were constructed from the teacher survey. We used factor and 
reliability analyses to determine the item consistency and reliability of the survey items that 
measure the same underlying construct to create the 13 teacher outcome scales. The reliability 
coefficients of all the scales are larger than .70. We also created three other simple count 
measures that counted the frequency teachers engaged in outreach, governance, and a variety of 
assessment activities. The 16 teacher outcome measures are listed in Exhibit 1-3. 
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Exhibit 1-3. Teacher Survey Outcome Factors 



School Culture and Professional 

Classroom Practices 


Learning Community 

• Learner-centered instruction 

• 

Collaboration around instruction 

• Assessment-informed practice 

• 

Trusting relationships between teachers 

• Regular use of a variety of assessments* 

• 

Effective principal PreK-3 leadership 

• Developmental^ appropriate practices 


Teacher Leadership 

• Emphasis on higher-order thinking skills 

• 

Teacher leadership* 

• Differentiated instruction 

• 

Governance activities* 

• Culturally responsive instruction 

• 

Early learning outreach activities 

• Family partnerships 

• Early childhood instructional knowledge 

• General instructional knowledge 


““Indicates a simple count measure 

No asterisk indicates the measure is a mean of responses to the multiple items in the subscale. 


Teacher Observations. Researchers conducted classroom observations of ECTLSI teachers and 
a group of matched comparison teachers located within control schools at two points in time 
using the Classroom Assessment Scoring System (CLASS). A baseline observation occurred 
early in the teachers first year of the program and a follow-up observation occurred after or near 
the end of the program. For the first cohort of ECTLSI students and their matched comparison 
teachers, baseline classroom observations were conducted in fall of 2011. For the second cohort 
of ECTLSI teachers and their matched comparison teachers, observations were conducted in fall 
of 2012. All follow-up observations were conducted in fall 2014. 

Exhibit 1-4 shows the total number of teachers observed at baseline and at follow-up. A total of 
49 treatment and 48 matched comparison teachers participated in observations at baseline. At 
post-intervention, researchers attempted to observe all teachers again. However, researchers were 
only able to conduct observations with 37 treatment teachers that remained in the program and 
25 comparison teachers. These teachers comprise our analytic sample. It is important to note that 
due to the differential attrition between treatment and control groups for the follow-up classroom 
observations, it is possible that the results from this analysis may be biased. For example, if those 
control teachers who did agree to be observed at follow-up are more motivated teachers than 
those missing their follow-up observation, our control group may appear artificially positive on 
classroom instruction practice as compared to the treatment group. 


Exhibit 1-4. CLASS Observation Teacher Sample 



T reatment 

N 

Control 

N 

Observed at baseline 

49 

48 

Observed at follow-up/Analytic sample 

37 

25 
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The CLASS has been extensively validated and can be used reliably in both research and 
evaluation studies, and as a tool for guiding professional development 2 . The measure gives 
scores on three domains of classroom quality that have been linked to student learning and 
achievement: emotional support, classroom organization, and instructional support. Each domain 
is composed of multiple dimensions, which are rated on a scale of 1 to 7, with a score of 1 to 2 
considered as low, 3 to 5 as middle, and 6 to 7 as high. 

Student Achievement Data. To examine the impact of FMTI on reading and math achievement 
for students in kindergarten to grade 5, we collected standardized test scores from the spring 
prior to the start of the program (spring 2011) and for each spring that the program was 
implemented (spring 2012, 2013, 2014). The school district administers the Stanford 
Achievement Test-Tenth Edition (SAT-10) to students in kindergarten through grade 2 and the 
Florida Comprehensive Achievement Test (FCAT) to students in grades 3 - 5. In the last year of 
the study, we obtained from the district data files containing the SAT-10 and FCAT scores for all 
students in grades K-5 ever enrolled in treatment and control study schools during the year prior 
to the start of the program and the three years of the study. For the schoolwide impact analysis, 
because both SAT-10 and FCAT has developmentally scaled scores that are comparable across 
grades, we standardized the 2013-14 outcome test scores by student baseline grade level in 2010- 
11; therefore, even if a student is retained in grade, he is still compared with his original cohort at 
baseline. For ECTFSI teacher impact analysis, since we only examined one-year progress of 
student achievement, we standardized the 2013-14 outcome test scores by student grade level in 
2013-14. For reading and math separately, we calculated z-score by taking the difference 
between each student’s original test score and the mean score for his cohort in the whole district, 
then dividing by the standard deviation of the score for his cohort. In addition to outcome scores, 
we also standardized pre-test scores using the same approach and included them as covariates in 
the impact analyses. 

Covariate student and school data. We collected data on student and school characteristics to 
use as covariates in analytic models to improve the precision of impact estimates. 

Report Overview 

Chapter 2 presents key findings from the formative evaluation, including information on the 
fidelity of program implementation. Chapter 3 presents findings about the impacts of FTMI on 
school culture and teacher practices, both schoolwide and for those teachers who participated in 
the ECTFSI graduate program. Chapter 4 describes the findings about the impact of FTMI on 
student achievement for students schoolwide and for those who had an ECTFSI teacher. A brief 
conclusion presents the key lessons learned from this study. Finally, the report includes technical 
appendices that provide more details about the study methods, the fidelity implementation, and 
the analytic models used to determine impact. 


Pianta, R. C., La Paro, K., & Hamre, B. K. (2008). Classroom Assessment Scoring System (CLASS). Baltimore: Paul H. 
Brookes. 

Cash, A. H., Hamre, B.K., Pianta, R.C., & Myers, S.S. (2012). Large-scale rater calibration for an observational instrument. 
Retrieved from: http://curry.virginia.edu/uploads/resourceLibrary/Research_BrieF-_Rater_calibration.pdf. 
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2. Formative and Fidelity Evaluation Findings 


In Years 1 and 2 of FMTI’s implementation, SRI conducted a formative evaluation to learn about 
the program’s implementation, accomplishments, challenges, and strengths to support program 
refinement, replication, and the sharing of lessons learned with the field of education. The 
formative evaluation collected data through interviews with district and school level staff, 
administrative data on program participation, and review of program documents. 

This chapter provides information on the implementation of the key program components—the 
job-embedded graduate degree program, the Teacher Fellows program, the Principal Fellows 
program, the Summer Leadership Institute, and program enhancements. 

Job-Embedded Graduate Degree Program 

At the heart of the Florida Master Teacher Initiative (FMTI) is the graduate program. Developed 
and administered by the UF College of Education, the graduate program has an early childhood 
specialization and specifically meets the needs of early childhood educators (grades pre-K-3) 
working in high-need communities. The FMTI graduate program is an enhancement of UF’s 
Teacher Leadership in School Improvement (TLSI) graduate program with an emphasis on early 
childhood (EC), so it is called the ECTLSI program. The ECTLSI graduate program blends 
online instruction with face-to-face pedagogy by a professor-in-residence who works alongside 
the teachers and administrators in the participating schools. The program is job embedded and 
designed to help teachers immediately put to use their theories, objectives, and learning to solve 
the challenges they face in their classrooms. 

The graduate degree program is a 2.5-year, 39-credit-hour program. It is divided into four terms 
a year (fall, spring 1, spring 2, and summer), and participants take one course each term except 
for summer, when they take two courses. In addition, each summer includes a three-day institute 
focused on developing participants’ facilitation and leadership skills. The ECTLSI program is 
serving two cohorts of teachers: Cohort 1 began the program in the summer of 2011 and Cohort 
2 began the program in the summer of 2012. Both cohorts followed the same sequence of 
courses and engaged in similar leadership and research opportunities. 

The goal of the ECTLSI graduate program is to improve teacher professionalism along three 
dimensions: instructional practice, leadership, and the ability to engage in inquiry to improve 
teaching and learning. By facilitating improvement in these three domains, the program aims to 
help their participants achieve the three pillars of “master teacher,” “teacher leader,” and 
“teacher researcher.” In the final semester of the program, teachers are expected to document 
their progress towards these goals by assembling a portfolio of their work including artifacts for 
each of the three pillars. 

Course Development and Adaptation 

The graduate degree program is a joint venture of two schools within UF: the School of Special 
Education, School Psychology & Early Childhood Studies and the School of Teaching & 
Learning. The program was adapted from an existing graduate program to include an early 
childhood focus. This adaptation involved the design of four new early childhood courses: 
Developmentally Appropriate Practice in Early Childhood Education, Assessment and 
Evaluation in Early Childhood Education, Family Involvement in Early Childhood Classrooms, 
and Policy and Transitions in Early Childhood. In addition, UF faculty redesigned existing 
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courses, such as Differentiated Instruction and Transforming the Curriculum, to create a stronger 
focus on early education by including more examples and texts relevant to the experiences of 
preschool through grade 3 classroom teachers. 

The course designers also built in mechanisms for improvement. After each course, students are 
asked to provide feedback that is analyzed by the teaching professor and the course designer. 
Moreover, UF faculty hired graduate students to interview graduate degree teachers from Cohort 
1 to learn about their perspectives on which features of the program had the largest impact on 
their practice and which features of the program could be improved. Where relevant, changes 
were made to courses before they were offered to Cohort 2. 

For the most part, both the professors and designers of the ECTLSI courses believe that they 
went well the first time around. In particular, the original courses required few adjustments after 
their initial adaptation to emphasize early learning content. UF faculty explained that these 
courses had already gone through several cycles of formative feedback and revision and were 
consequently already functioning smoothly. The newly developed early learning courses also 
were largely successful, although they required some more substantial revisions than the original 
course sequence. For example, in their feedback for the Developmentally Appropriate Practice 
course students expressed concern that the course was overly focused on very early childhood 
(birth through preK) whereas many of them teach kindergarten through 3 ld grades. In response, 
the course designers reduced some of the early childhood readings and introduced additional 
videos of primary grade children. Of all of the courses, Assessment and Evaluation in Early 
Childhood Education was the least well received by Cohort 1 students. For Cohort 2, UF staff 
heavily re-organized the assessment course and used a different professor, who was more 
experienced with the ECTLSI students, to teach it. By all accounts, the course went more 
smoothly for Cohort 2. 

Through experimentation while teaching Cohort 1, UF faculty also refined the blended model for 
delivering instruction. An unsuccessful experiment with online-only delivery for one course 
confirmed the need for all professors to include at least some face-to-face instruction. UF staff 
also learned the necessity of allowing adequate time for students to become familiar with new 
online technologies before requiring their use in class. For example, program staff added “voice 
thread” to the online platform mid-way through the program, which allows a discussion thread 
using recorded video messages and presentations. This addition was meant to enable more 
compelling conversations and sharing than a purely written dialogue; however, teachers 
struggled with learning the new technology while learning new content at the same time. Once 
teachers became comfortable with the voice thread feature it ceased to become a barrier. For UF 
staff, this experience highlighted the importance of respecting the learning curve associated with 
introducing new technology for teachers who already have a lot on their plates. 

The final category of revisions UF staff applied to the ECTLSI courses grew out of an increased 
understanding of participants’ roles as full-time teachers and graduate students working under 
the constraints of the public school calendar. As one faculty member explained, 

As we see how students respond to coursework, and when they really start 
to grow with coursework and assignments, things that feel too theoretical 
have been pared down, I’ve watched that happen. So I’ve seen things like 
the amount of scholarly reading has been reduced, and the amount of 
conversation and connection to practice has increased. 
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Likewise, other professors described an increased awareness among UF faculty that ECTLSI 
participants could not be considered “traditional” students in a lot of ways and that the goals of 
the program “are to strengthen teachers’ knowledge and practice, we aren’t preparing them to be 
researchers at an R-l university.” After teaching Cohort 1, professors also realized that some of 
the due dates for their assignments coincided with particularly stressful times during the public 
school calendar such as during the high-stakes state testing period. In response, UF professors 
shifted due dates for Cohort 2 to be more accommodating of their responsibilities as teachers. 

Recruitment of Graduate Students 

In contrast to the successful process for course development and adaptation, the recruitment of 
graduate students was a challenge throughout the life of FMTI. UF professors-in-residence spent 
considerable time and effort recruiting teachers for the graduate’s degree program but were 
unable to meet recruitment goals for either Cohort 1 or Cohort 2. In addition to preparing 
materials describing the program, recruitment strategies included visiting each school to discuss 
program structure and benefits with the principal and other staff, having current or past UF job- 
embedded graduate degree students present information about their experiences in the program, 
producing a video that promoted the ECTFSI program through student testimonials, and holding 
informal information sessions in local restaurants where prospective candidates could learn more 
about the program. 

Understanding why teachers do and do not apply could help contextualize the disappointing 
participation statistics and inform future recruitment efforts for FMTI or for similar initiatives. 

As discussed in detail in the 2012 FMTI formative report, the most frequently cited reasons that 
Cohort 1 teachers applied for the grant include the fact that the program would be free to 
participants, the good reputation of UF, the ease of completing the program online, its immediate 
applicability to their instructional practice, and a personal drive to become a better teacher. 
Cohort 2 teachers cited many of these same factors when explaining their decisions to apply. For 
example, as one Cohort 2 teacher explained, 

I wanted to enhance my practice... I wanted to be a better teacher... not 
just a better educator. I wanted to be a better leader. 1 knew this 
program — researcher, leader, and master teacher. 1 just wanted to 
improve on my craft. I've been teaching 16 years. I don't want to say I was 
in a rut, but I wanted to take it to a different level. 

Cohort 2 teachers, while echoing many of the same reasons for applying to the ECTFSI program, 
were additionally able to shed some light onto why they applied the second year the grant was 
offered, but not the first. For example, one teacher explained that the program was not advertised 
as much for Cohort 1 as it was the year she joined. When recruitment was underway for 
Cohort 1, she was teaching 4 th grade, making her ineligible for the program; however due to the 
more intensive advertising the following year she asked her school administration if they would 
move her to 3 rd grade if she applied for the program, which they did. This example both 
underlines the importance of aggressive advertising as well as providing some validation for UF 
staffs concern that their recruitment strategies did not effectively target early childhood 
teachers. 

Several other Cohort 2 teachers cited their exposure to the program during its first year of 
implementation—either through participation in the Teacher Fellows program or simply through 
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hearing about the positive experiences of Cohort 1 teachers—as a motivator for applying for the 
second round. For example, one teacher said, 

At first I wasn 7 sure, I was getting married and I was on the fence, you 
know it’s a lot to take in. But [the first cohort] had so many positive things 
to say about it, they were so energetic and motivated that I was like “I 
want to be that, I want to extend my thinking in terms of education ”... They 
were honestly the ones that were like '‘do it, because it changes your life. ” 

Similarly, a principal from one of the participating schools described the process as: 

An evolution, those high-flyers that we have took to it recdly well and [the 
other teachers] see the things they’ve brought to our school and the 
empowerment and camaraderie they’ve had, and they see how wonderful 
the UF people have been, and then people gravitate towards that that 
want to be a part of something like that. 

Understanding why teachers apply to the program can inform future recruitment, but it is equally 
informative to understand the barriers that keep them from applying. For both cohorts, the GRE 
was one of the most difficult dilemmas for program developers at UF. Another challenge is the 
university’s expectation for applying graduate students to have earned a 3.0 grade point average 
(GPA) in their undergraduate studies. As a land grant institution, part of the UF mission is to 
work with elementary schools. On the other hand, as a competitively ranked public research 
institution, UF needs to maintain strong GRE and GPA scores to maintain its standing. Although 
FMTI cannot waive the GRE or GPA requirements altogether, it does assist candidates in 
overcoming these barriers. For example, FMTI provides materials and test preparation courses to 
help candidates pass the exam. It also devised multiple admission plans to facilitate admissions 
for candidates with weak GRE scores. Plan B is a conditional admittance for those candidates 
with borderline scores but who had a sufficient undergraduate GPA. They are accepted to the 
program with certain conditions, one being that they must maintain a B average for their first 
courses. Plan C is for candidates who need higher GRE scores before they can be admitted. Plan 
C candidates are allowed to take the first course as non-degree-seeking students while they 
simultaneously take a rigorous GRE preparation course. Then they have to retake the GRE. Even 
if their scores remain borderline, it is easier to argue that these candidates will be successful in 
the program after they demonstrate success in the first courses. Ten teachers took advantage of 
the Plan C opportunity (two in cohort 1 and eight in cohort 2); which yielded six teachers fully 
accepted into the ECTLSI program. 

Aside from the GRE requirement, some teachers did not apply because they already had a 
master’s degree. Although FMTI does offer a “specialist” degree for participants who already 
have a master’s degree, many teachers cited the combination of already having a master’s degree 
along with having a lot on their plate due to work and personal obligations as their explanation 
for not applying. A principal at one of the schools with no ECTLSI graduate students explained 
that her “biggest challenges were that 50% already have a master’s degree, and the other 50% 
did not want to take the GRE. They were afraid or discouraged through the process.” 

UF faculty were especially disappointed that they were not able to meet their recruitment goal 
for Cohort 2, given that they had both more time and more experience to improve upon their 
efforts from the first round. Several UF staff members speculated that they may simply have 
overestimated the demand for their program from any given school, saying that they had “gone 
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back to the well too many times.” By contrast, one principal believed that if the program 
continued, his school would continue to field new applicants for the program as his older staff 
turned over and new teachers joined. From this perspective, the graduate degree model might not 
have overestimated demand all together, but rather overestimated the size of cohort that could be 
recruited in any given year. 

Another issue with recruitment UF faculty identified was the inability to establish a foothold in 
some schools. Although the schools applied to be part of FMTI, seven schools ended up with no 
ECTLSI graduate students. Several UF faculty surmised that these schools, which included 
transformation schools, may have been pressured to apply by their regional superintendents. 

Also, they recognized that these schools are already involved in many other school improvement 
initiatives, which makes it a challenge for them to engage in FMTI programs as well. Program 
staff speculated that recruitment may have been less challenging if they had made administrative 
commitment to the program the central criteria for selecting schools. One UF faculty member 
reflected: 

I think it’s really important that you select schools that truly want to do 
what you want to do. I’m not sure if we did that right from the beginning, 1 
think there was a lot of care about selecting schools from each region, and 
for political reasons, each school board member had a certain number of 
participating schools. It was very careful, but for me the more important 
thing is—do you [the principal] agree with the model conceptually and 
are you recdly going to push it and make it work? 

Similarly, a principal of one of the participating schools argued that the program might have 
gained better traction in schools if they had more effectively enlisted principals in their efforts. 

Principals need to be more involved in understanding what this really 
means so they can be your best ambassadors and your best support 
system. 

Retention of Graduate Students 

A total of 61 teachers (37 in Cohort 1 and 24 in Cohort 2) representing 13 of the 20 treatment 
schools were admitted into the ECTLSI program. By the end of Year 4, 30 Cohort 1 and 18 
Cohort 2 teachers had graduated the ECTLSI program. Thus, 48 of the original 61 teachers 
(79%) had graduated and were retained until the end of the program. 

Fidelity for the ECTLSI Program 

By the end of their program, ECTLSI graduate teachers are expected to maintain a 3.0 GPA, 
complete scheduled classes, attend a facilitator training, engage in formal inquiry projects, and 
facilitate a Teacher Fellow group, other inquiry group, professional learning community (PLC), 
or other professional development activity. Data collected show that most teachers who remained 
in the ECTLSI program did meet the program expectations (Exhibit 2-1). However in Years 2 
and 3, only 10 schools had at least two teachers in the program who could work together on 
coursework and school leadership and research efforts, an important school-level fidelity 
criterion. Because only 10 of 20 schools had at least two teachers enrolled in the ECTLSI 
program by Year 2, there was low fidelity for the ECTLSI program overall. 
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Exhibit 2-1. Teacher Progress in ECTLSI Graduate Program 


Elements 

Percent of 
ECTLSI 
Graduate 
Teachers 

Year 1 

Percent of 
ECTLSI 
Graduate 
Teachers 

Year 2 

Percent of 
ECTLSI 
Graduate 
Teachers 

Year 3 


23 

II 

CO 

^1 

23 

II 

^1 

23 

II 

-P** 

OO 

Maintained a 3.0 GPA 

100 

100 

98 

Completed scheduled classes 

100 

100 

98 

Completed an inquiry (ever) 

93 

100 

100 

Participated in facilitator training (ever) 

83 

83 

100 

Facilitated a Teacher Fellow or other 
professional development group in their 
school (ever) 

77 

55 

91 


Program Strengths 

ECTLSI graduate students across both cohorts were overwhelmingly positive about the program. 
The three most widely cited program strengths were (1) the applicability of the course content to 
the participants’ instructional practices, (2) the collaboration it engendered both within and 
across schools, and (3) the support the students received from UF faculty, both the professors-in- 
residence and Gainesville-based faculty. 

Applicability. The ECTLSI program was explicitly designed to be job-embedded and to 
encourage teachers to immediately apply what they learned through their coursework to their 
own classrooms. As one professors-in-residence explained, “If you are not seeing the value this 
course has to your practice and it’s not immediately impacting your practice, then I’m not doing 
a good job making that connection for you.” This strong emphasis on applicability was 
extremely well received by teachers across both cohorts. For example, one teacher praised the 
way the professors facilitated discussions of the course readings, focusing on helping teachers 
adapt what they learned to their own situations. 

Several teachers pointed out how the applicability of the coursework was different from their 
original education training or other master’s degree programs. One teacher explained, 

When I did my [first] master’s, I did so many things and none of them 
really applied to what I was doing. It gets annoying. The advantage of this 
program is getting to do things in class that I can implement in my own 
classroom, it makes it fun and easier to do work and do my assignments. 

Similarly, a teacher from a different school said, 

When you go to school to learn to be a teacher, you ’re learning all of 
these things but you don ’t put it into practice until you become a teacher. 

So this is amazing, you just put it in, OK it doesn 7 work for me, alright 
let’s try something different, and it’s all within the class and you ’re also 
getting feedback from your classmates. 

Indeed, several teachers referenced the combined benefit of the ECTLSI program’s applicability 
and the collaboration it engenders. These two program features mutually reinforce one another 
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because teachers are not only able to immediately apply their learnings to their own classroom, 
but they also are able to get immediate feedback by comparing their strategies with similar or 
different approaches employed by other colleagues in the program. 

Collaboration. The collaborative nature of the ECTLSI program both encourages teachers to 
learn from one another and helps them to feel less isolated. In schools with more than one 
participant, many teachers reported working closely together in conducting their inquiries, 
completing projects and other assignments for their classes, and collaborating on their 
instructional plans and strategies. For example, one teacher explained: 

It’s really rewarding, it’s motivating, you ’re not just by yourself in the 
world. With this program you ’re not alone, everybody is together and 
we ’re all having the same issues. “This is how / dealt with it, how did you 
deal with it? Maybe I should try that. Maybe you should try what I did. ” 

And I love that, definitely being part of a team, and that is what education 
is, we are all a team trying to help our students. 

In one school with a high level of participation in the ECTLSI program (two teachers in Cohort 1 
and six teachers in Cohort 2), the participants took the initiative to start their own professional 
learning community dedicated to offering each other support in their graduate work and 
strategizing how to translate what they learned in the program to improvements in schoolwide 
teacher morale. One of the teachers from this school emphasized the importance of the cohort 
aspect to her overall experience: 

You could just take the online course and do the work, but it’s not the 
same as doing it with a cohort. Fortunately we have a large cohort of 
participants in the master’s program in this school, we're constantly 
collaborating about things we need to do for class or reflecting on things 
we've done in class. And it’s easy to do because we're all here together. 

In addition to the benefits of interacting with ECTLSI colleagues within their own schools, 
several teachers also reported their appreciation of the opportunities the program provides for 
connecting with colleagues across school boundaries. One teacher explained how the online 
forum allowed the teachers in her school to “communicate with other professionals, to give our 
opinion, and get their opinion.” Another teacher explained that even though they are often only 
required to respond to two teachers’ posts online, “you want to go back and see what all of them 
have to say, and sometimes you want to respond to more than two. I love reading what they 
respond back to me.” 

Support from UF faculty. The third strength of the program is the support ECTLSI graduate 
students receive from the professors-in-residence and other UF faculty. They reported that the 
professors are available and accessible and that they provide meaningful feedback and support 
for meeting program expectations and requirements. Teachers described the professors as 
“friendly and easy to contact,” “understanding,” and “empathetic.” One teacher favorably 
contrasted her experience with the professors-in-residence to her undergraduate experience: “It’s 
not just like you’re just another face, you’re just another number in passing... sometimes in 
college that is what I felt.... No, they meet you, they get to know you, they embrace you, they 
nurture you.” 
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When asked what characteristics allowed them to do their jobs so effectively, both professors-in- 
residence cited their experience teaching in the Miami public schools as critical to their ability to 
translate course content to the context in which the ECTLSI teachers are operating. ECTLSI 
graduate teachers echoed the importance of their professor’s prior teaching experience when 
discussing the strengths of the program. One teacher explained, “they are coming from a 
situation that is similar so they are relatable.” Another teacher emphasized: 

They were there, so they understand what it is to be working as well as 
doing your master’s. When implementing, sometimes you might not have 
the environment to do that and they will say, “You know what, let’s adapt 
it for you, let’s make it so it works for you in your environment. ” They ’re 
not just like “deal with it. ” They say, “Let’s change it up so you ’re kind of 
doing the same assignment but within what you have, what can you do 
with the group that you have. ” They are just amazing. 

Program participants also were clear that the professors-in-residence were available to help with 
any issue, whether it was directly related to the coursework or a more personal issue that was 
interfering with their coursework. A few teachers had major life events or family emergencies 
take place while they were in the program. They explained that the flexibility and supportiveness 
of their professors helped them get through those periods successfully. 

Program Challenges 

All teachers in the ECTLSI graduate degree program spoke highly of the program and its 
strengths. A few, however, did bring up challenges. A common challenge was finding time to 
complete assignments, especially with other school and family obligations. But despite this 
frequent complaint, teachers largely described the program as “doable,” despite how much they 
had on their plate. 

Being designated a school targeted for improvement. As part of its improvement strategy 
under No Child Left Behind, the Miami-Dade County School District established the Education 
Transformation Office (ETO) to improve test scores in its lowest performing schools. Teachers 
working in schools operating under the oversight of the Education Transformation Office (ETO) 
reported more serious challenges to implementing the new instructional practices learned in their 
courses. As of the 2013-14 school year, 10 of the 20 FMTI schools were identified as ETO 
schools because of their persistently low achievement on state standardized tests. One of the 
professors-in-residence explained that the teachers in these schools are under such pressure, that 
she cannot push them to the same extent as other teachers: 

But then you have the ETO students that you have to be very careful 
with...those you try to support more on the level of inquiry. For example, 

“well if you are going to have to do that anyway, why don ’t you do an 
inquiry on it? ” You don’tpush them as a teacher leader because they are 
ready to burst. You try to push them as master teachers and teacher 
researchers. 

Although ETO oversight comes with extra supports for schools such as additional coaches, some 
teachers reported that the school climate under ETO was not conducive to the type of 
experimentation advocated by the ECTLSI program. “Everyone is walking on eggshells,” one 
teacher explained. As a teacher from a different school described “coaches [are] all over the 
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place. And you’re being told today is A, and tomorrow is B, and then the next day is C— it’s a 
lot.” A third teacher from yet a different ETO school concluded, 

The biggest barrier in this school is that we're ETO. Since our grade was 
low performing, we have a lot of restrictions. We're very much scripted to 
what we need to do, at what time, and how. It’s hard to follow those rules 
and still incorporate the building classroom communities and the feeling 
of a family when you have 10 minutes for this, 15 minutes for that, and 
let’s move. We're under a lot of scrutiny. 

When the challenges with ETO schools began to surface, UF faculty worked with one of their 
doctoral students, who runs the elementary division of ETO, to help teachers understand that the 
ECTLSI program and the goals of ETO can be mutually reinforcing. One Cohort 2 teacher 
specifically mentioned that this effort helped assuage some of her concerns: 

Initially it was [challenging], because we didn't know how to channel 
what we learn into what ETO is expecting of us, but after talking to other 
teachers in ETO schools and even after the talking to the ETO supervisor, 
who came out to one of our classes and spoke to us and told us it wasn't 
something separate and apart. She's in the UF program too. She was 
saying how everything can be incorporated, it's not one thing versus 
something else... I don't think it's challenging anymore. I did feel it before, 
but now I don’t. 

A Cohort 1 teacher from the same school discussed how one of the recent ECTLSI courses she 
took helped her implement the practices required by ETO: 

Now that we are under ETO they are requesting differentiated instruction, 
so everything has fallen into place. 1 just took that course last semester, so 
I am able to implement that and, because of the course, we were able to 
put the planning... I know the reason WHY it works, and now I am able to 
put it in practice because I have the tools. 

Conflicts with principal policies and goals. Another challenge mentioned by some teachers 
was the conflict between school policies set by a principal and the some of the practices 
promoted by the ECTLSI program. For example, the practices encouraged around family 
engagement sometime conflicted with school policies. One teacher described, 

What I want for this year is to have my parents impact the classroom 
more, use their funds of knowledge, and having them be part of their 
children’s education. But we sort of have a closed campus, which doesn 7 
really invite parents unless we do a big activity, but we have to jump 
through hoops to get those things done. 

Similarly, some teachers and program staff mentioned lack of principal buy-in or full 
understanding of the purpose of FMTI in certain schools being a challenge. For example, one 
teacher explained, 

I would say that there is a lack of knowledge about the true purpose of the 
program ...because at times we’ve been directed with our professional 
learning communities topics that we should discuss but this is not what we 
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are doing, the purpose of the PLC is not to support the administrator’s 
agenda. 

Teacher Fellows Program 

FMTI seeks to improve the practice of all teachers in its program schools, not just participants in 
the embedded ECTLSI graduate degree program. The Teacher Fellows program provides a 
means to reach a broader range of teachers at a less intense and more accessible level than the 
graduate degree program. Over the course of a year, a Teacher Fellow facilitator, often a teacher 
in the ECTFSI graduate program, leads a group of teachers at a school through a guided inquiry 
into their teaching practice. Supported by the facilitator and their peers, participating teachers 
pose questions or “wonderings” about various areas of student learning that they feel could be 
improved, select a specific student learning area to improve, research potential strategies for 
doing so, implement a strategy with their students, assess the impact by collecting and analyzing 
data, and share the results of this inquiry with their peers. The program culminates in a 
presentation of the inquiries at the end of the school year at the district-wide Fearning Showcase. 
Teacher Fellows earn a $400 stipend and professional development credit hours for participation. 
The Teacher Fellows facilitators receive training in facilitation, support, and materials from UF 
staff, and a $500 stipend. 

The Teacher Fellows program has two main components: 

• Teacher Fellows professional learning community (PFC) meetings to support teachers in 
developing, implementing, and reflecting on their inquiries 

• The Fearning Showcase, which both acknowledges the importance of teachers’ inquiries 
and enables them to share their learning and promising practices with teachers and 
administrators from other FMTI and Miami Counts schools. 

Teacher Fellows PLC Meetings 

Most schools ran Teacher Fellows programs and held the required six program sessions, with 
groups meeting formally about once a month every year. Only one school of the 20 treatment 
schools did not participate in 2011-12 and two treatment schools did not participate in 2012-13. 
All the treatment schools participated in 2013-14. A total of 235 teachers participated in the 
Teacher Fellows program in Year 1, 242 teachers in Year 2, and 320 teachers in Year 3 (this was 
an average of 30-45% of faculty at these schools). 

Most ECTFSI teachers participated in the Teacher Fellows program and often served to lead the 
groups, serving as Teacher Fellows facilitators. Non-ECTFSI teachers provided a variety of 
reasons for participating. The most common reasons for participating were prior positive 
experiences with action research and inquiry, the opportunity for collaboration with other 
teachers, and the opportunity to examine their own practices. A teacher fellow explained why the 
Teacher Fellows program was of interest to her: 

I find the PLCs more helpful than traditional professional development 
because we [teachers] all have a lot of expertise, we’ve been teaching for 
a long time. It’s the ability to share our expertise, and knowledge, and best 
practices that work that we can take or reject. With a professional 
development you are supposed to be just a recipient of these ideas and 
implement them and it doesn 7 work traditionally. 
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Learning Showcase 

The Learning Showcase is a one-day districtwide conference held in May that provides an 
opportunity for K-8 teachers, early education teachers, principals, assistant principals, and 
community involvement specialists to present their inquiry projects to the wider district 
community. The event acknowledges the efforts of educators to improve their practices and 
provides an opportunity for educators to share best practices and new innovations in education. 

In May 2014, UF and the district hosted the sixth Learning Showcase in Miami, and the third 
Learning Showcase to involve FMTI schools. The showcase also featured the inquiry projects of 
educators in from other schools, including schools participating in a related initiative, Miami 
Counts. In addition, district administrators, including the superintendent, attended the showcase 
and many of the presentation sessions. 

Of the almost 450 attendees at the 2014 Learning Showcase, 294 of them were teacher fellows 
from FMTI schools. While teachers made up the largest percentage of participants, other staff 
such as paraprofessionals, community involvement specialists, and administrators from FMTI 
schools presented as well. 

The showcase presentations focused on a broad spectrum of topics, including language arts, 
math, science, art and technology, early childhood education, student engagement, school 
culture, special education, and leadership. 

The majority of teacher fellows interviewed reported the Learning Showcase to be a positive 
experience. Teachers spoke positively about the opportunity that the Learning Showcase 
provided to learn from and share with their peers. Teachers below described their experience at 
the Learning Showcase: 

I got to see that something works. You go to workshops and you ’re always 
given information... [and] everything looks great... but [you think,] “Yeah, 
but not with my kids. ” But it actually works with all levels... of kids... 1 
like having the sessions because you get so much from different teachers. 

You hear so many different perspectives on things and then you can take 
that back and say, “I’m going to try this, I’m going to try that, or I’m 
going to do that differently. Maybe it’s not working because of this. ” 

I love going to the showcase and learning about what everyone else is 
doing. I always walk away with a whole bag of goodies. 

However, some teacher fellows provided mixed or negative feedback on the showcase. Negative 
feedback included reports that several of the presentations or inquiries were of low quality or 
were not useful to inform their own practices. As one teacher said, “The other presentations I 
went to were really disappointing.... The quality of the research and the inquiry itself was not 
good.” 


Fidelity of the Teacher Fellows Program 

The Teacher Fellows program was conducted with medium or high fidelity all three years in the 
vast majority of treatment schools, with a growing number of teachers participating each year 
(235 in Year 1, 242 in Year 2, and 320 in Year 3). This was an average of 30-45% of faculty at 
each of the schools. The vast majority of teachers who participated in the Teacher Fellows 
program showed active and consistent engagement. For example, in Year 3, 86% of Teacher 
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Fellows attended all six sessions, 92% presented at the Learning Showcase, and 90% wrote and 
submitted a summary of their inquiry (Exhibit 2-2). Of the teachers interviewed, only two 
mentioned challenges related to completing their inquiry project and both of them said their 
challenge was related to finding the time needed to complete the project. 


Exhibit 2-2. Participation in Teacher Fellows’ Activities 


Activity 

2011-12 

2011-12 

2012-13 

2012-13 

2013-14 

2013-14 

Number 
(n= 235) 

Percentage 

Number 
(n= 242) 

Percentage 

Number 
(n = 320) 

Percentage 

Attended all six sessions 

219 

93% 

228 

94% 

275 

86% 

Presented at Learning 
Showcase 

217 

92% 

238 

98% 

294 

92% 

Submitted written 
summary of inquiry 

220 

94% 

240 

99% 

288 

90% 


Data sources: Teacher Fellows sign-in sheet for 2011-12, 2012-13, and 2013-14; administrative data of Teacher 
Fellows receiving credit and receiving a stipend for participating. 


Program Strengths 

The majority of teachers spoke positively of their experience in the Teacher Fellows program. 
Nearly all interviewed teachers had participated in the program for two years and planned to 
participate again. The aspects of the program that contributed to teachers’ positive experience 
included the layers of support embedded in the program, the clearly structured process and 
materials available for leading teachers through the inquiry process, and the protocols and time 
for collaboration. 

High-quality training and facilitation. High-quality training and facilitation was important for 
supporting successful Teacher Fellows programs. The Teacher Fellows program is structured so 
that Teacher Fellow facilitators receive training in facilitation and support from UF staff, and the 
teacher fellows receive support from the facilitators and their peers. A Teacher Fellow facilitator 
said, “I’m the one that is going to support everyone.. .and say ‘You can do it. If you have any 
problems, I’ll try to help you.’ Just to be that supportive person.” 

This tiered structure creates an environment where all participants have resources to draw on as 
they work together on their inquiry. Reflecting on their past two years as a Teacher Fellow 
facilitator, a few teachers recalled initially feeling “intimidated” or “nervous” in the role. 
However, the training and support from the professors-in-residence supported the Teacher 
Fellow facilitators throughout the process. Furthermore, a few teachers mentioned the 
importance of trainings in developing their skills to use meeting protocols and effectively run the 
Teacher Fellow facilitator meetings. One Teacher Fellow facilitator described the importance of 
the coaches’ training played for her: 

The coaches ’ training was a huge thing. I didn't know what the protocols 
were before that and with that we learned community agreements, how to 
build the relationships with the teachers. 

UF bolstered the support this year by increasing the length of the training and planning session 
for Teacher Fellow facilitators from two afterschool sessions to two half-day sessions. Teacher 
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Fellow facilitators also valued having a co-facilitator at their school to engage in the work 
together. 

Clearly structured process and materials. The Teacher Fellows program provides a clear 
structure for leading teachers through an inquiry project over the course of six sessions. Each 
session has a clear focus, and Teacher Fellow facilitators are provided materials to support them 
in leading these sessions. The Teacher Fellow facilitators receive sample agendas, protocols, 
suggested activities, and readings for each of the meetings. UF strengthened the materials for 
Year 2 by tailoring them to the local context and providing examples specifically from Miami. 
Teacher Fellow facilitators reported that having all of these materials helped them feel prepared 
to lead the Teacher Fellows program. One Teacher Fellow facilitator also noted having all the 
Teacher Fellows materials available electronically in Year 2 made it much easier for them to find 
and adapt forms, protocols, and other materials for their Teacher Fellow meetings. 

Protocols and time for collaboration. A key aspect of the Teacher Fellows program is the 
opportunity for teachers to share their instructional practice and learn from each other in a 
supportive environment. The structure of the program, including the collective establishment of 
group norms and use of protocols to guide conversations explicitly tries to develop this 
environment. One initiative leader summarized the benefit of providing time and structure to 
support deep professional conversation among teachers: 

Providing teachers with a formed opportunity for meeting, talking and 
learning from one another is very powerful. As teachers we often walk into 
the building, nose to the grind and that 's it before we come up for cur. 

Because of demands, some of the conversations aren't deep enough and 
become venting session... [Gjiving them tools to have a conversation that 
it doesn't become a venting session, and giving them a frame to think 
about their craft and improve whatever they decide, that mechanism is 
very powerful. I think that's why it's so successful. 

One teacher articulated how the Teacher Fellows program created a space in which teachers’ felt 
safe to share and engaged with their peers: 

Teacher Fellows brings the school together. You can speak and don’t need 
to worry it will leave the room, and everyone is actively listening. 

Teachers don 7 like to share, but UF promotes norms that it’s okay to 
share. 

Teachers also appreciated having specific time set aside to meet as professionals, given the 
challenges of finding time within the school day to talk with peers. One teacher described this 
challenge: 

There is a designated time for collaboration because you know as a 
teacher it’s almost impossible to do that. You 're in your classroom with 
kids and this notion that we are all going to get together and collaborate, 
well fine, just pay me an extra day and Fll collaborate with the teachers. 

Our job is to spend time with our students. So having a designated time to 
collaborate is definitely a plus. 
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Program Challenges 

While comments about the Teacher Fellows program were generally positive, teachers and FMTI 
staff noted several challenges. 

Some inquiries were of mixed quality. In year 2 of the program, some teachers noted that not 
all inquiry projects were of high quality. This emerging theme may reflect higher expectations or 
a deeper understanding of inquiry as the program matured. One teacher believed that some 
teachers did not present true inquiries into their practice: 

I would say that the only thing that l was a little on the fence about was 
that some [presentations] were not inquiry based, it was more like a 
presentation. They just showed us what they were doing in their classroom 
but there was no, “This was my question, this is what I implemented, this 
was my result. ” It was more like, “This is what we do in our school. ” 

A few initiative leaders echoed the observation that some of the inquiry projects were weaker 
than others. Several hypothesized that the stipend attracted some teachers more interested in the 
money than engaging in a deep inquiry; however, they believe teachers still benefit: 

The quality of some of the inquiries ...is really great, and sometimes not so 
great.... The majority is good, [but] for people who want $400, it might 
not be of good quality. At the end of the day I want more quality, but the 
fact that you sit in six meetings talking about your practice, at the very 
least...if we increased... discussion that’s great. I feel it is giving teachers 
the idea that research is not scary... and you have power in your 
classroom. 

Like everything you have your bell curve. There are some teachers who 
are doing their action research and they ’re hying to find something that 
will help them, and there are some that are a little more superficial. 

Perhaps they don 7 get it or something else. In all, the process and the 
platform does help everyone, the fact that they get to turnaround and 
share with colleagues, they find that voice. 

Impacts were limited to participating teachers. Another challenge for the Teacher Fellows 
program is the difficulty in affecting teachers not participating in the program. The impacts of 
the Teacher Fellow program remained primarily within the program participants. Spillover 
effects were restricted due to a lack of opportunities for teacher fellows to share their inquiries 
with other staff and a school culture of isolation. Two participants talked about these limits: 

The ones that participate, [Teacher Fellows] has gotten us a little closer... 

It gives us a bond, just by meeting during fellows meeting. That’s why I 
requested that we try to get whole staff involved. 

The teachers more involved in the activities have more of a family feel, 
more comfy going to each other for advice, but more than half aren't 
involved. 

When talking about the differential impact on collaboration across grade spans, one teacher 
implied that pre-existing norms around collaboration partially explained the differences in 
participation and impacts: 
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Collaboration is a really positive change because of the Teacher Fellows 
program. Kindergarten and 1st grade teachers are all involved.... Those 
two grade levels work really well together as a team. They have good 
communication. They have less fighting. [I] can’t say the same things for 
upper grades that are less involved. Their collaboration not as strong. 

In addition, opportunities to share inquiries with peers within schools were limited. At the end of 
Year 1, some teachers mentioned plans to share inquiries the following year; however, most of 
these plans fell through. One school provided time for teachers to present their inquiries to their 
peers by recreating a mini-showcase within the school. In most schools, however, teacher fellows 
did not have a venue to share their work with teachers who did not participate in the program. 

The time commitment was sometimes a hardship. A few teachers mentioned the time 
commitment associated with participating in the program as a challenge for completing the 
inquiry and recruiting teachers to the program. Finding alternative afterschool childcare and 
needing additional time beyond the Teacher Fellow facilitator meetings to work on their 
inquiries with their group were the specific reasons mentioned. 

Principal Fellows Program 

The Principal Fellows program was designed to support principals in adopting a facilitative 
leadership approach and to enhance their ability to effect change within their schools. The 
Principal Fellows program has included several components. 

• The Principal Professional Learning Community (PPLC), which holds meetings of the 
principals across the 20 treatment schools facilitated by UF faculty 

• The statewide Principal Leadership Institutes, which afford the opportunity for principals 
to collaborate and share leadership practices with principals from UF’s network of 
partner schools across Florida 

• Inquiry projects in which principals can work with each other, with teachers, or on their 
own to study the effectiveness of new leadership and instructional practices, and 
opportunities for principals to present their what they have learned at the Learning 
Showcase. 

The ways in which the three components were implemented changed each year based on 
feedback from participating principals and the district’s calendar. 

Principal Professional Learning Community Meetings 

The frequency, focus, and membership of the Principal Professional Learning Community 
(PPLC) meetings have changed over the past 3 years. In 2011-12, the PPLC met locally five 
times during the school year. At the end of Year 1, principals provided feedback to UF faculty 
that there were too many meetings. Also, it was difficult to attend meetings after school when 
principals often had to address pressing issues that had arisen during the day. In response, the UF 
team reduced the number of meetings for 2012-13 from five to four, increased their length from 
2 to 4 hours each to allow deeper conversation, and held them in the morning rather than after 
school to avoid unexpected conflicts. For 2013-14, UF held three PPLC meetings rather than 
four because the district professional development calendar could not accommodate more. 
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The focus of the PPLC meetings also changed between Year 1 and 2. The first year of the PPLC 
focused primarily on having principals learn about and personally experience the inquiry 
process. All principals were required to conduct an inquiry project and many principals shared 
their inquiry work at the Learning Showcase. Even though principals reported that the inquiry 
projects had been valuable, they let UF faculty know that these projects were too much work on 
top of their heavy workloads. The principals also requested that Year 2 PPLC meetings include 
more content and resources, such as a book study on appreciative leadership. In response, the UF 
team decided to make inquiry projects optional and to make the PPLC into more of a learning 
community that uses the UF protocols to share dilemmas and ideas for how to solve them. In 
particular, the UF team introduced the protocol of “principal talk.” This protocol involves having 
different principals host each PPLC meeting. The host principal brings in data or a dilemma and 
shares it with the other principal fellows. The protocol guides the group through an analysis of 
the issues and possible solutions in a safe and supportive environment. 

Finally, the regional or districtwide composition of the PPLC meetings changed from year to 
year. In Year 1, three PPLC meetings were held by region and two brought all of the principals 
together as a single group. In Year 2, the UF team held only regional PPLC meetings and 
combined FMTI principals with principals from the districts’ 26 Miami Counts partner school 
sites in their regions, who were also receiving professional development from the same UF team. 
For the 2013-14 school year, the UF team has moved away from regional meetings to holding 
only centrally located meetings with all principals from the 20 FMTI schools. 

Statewide Principal Leadership Institutes 

Statewide Principal Leadership Institutes are an opportunity for principals to get away from their 
daily work to focus on learning and sharing practices with principals from other schools and 
districts. There were two statewide institutes in 2011-12, but only one in 2012-13 and 2013-14 
because of conflicts with the district professional development calendar. The institutes in 
2011-12 were held in two different Florida locations: Naples and Jacksonville. The focus of the 
first institute was on inquiry and appreciative leadership and the second statewide institute 
focused on cognitive strategies to promote student learning and engagement. 

In 2012-13, one statewide institute was held in May in Tampa, and included school 
administrators and district personnel from Hillsborough County Public Schools. The focus was 
on effective leadership and teacher evaluation. Participants learned about Hillsborough’s peer- 
supported teacher evaluation model through a panel discussion and school site visits. The group 
also learned about best practices in teacher evaluation from national expert, Dr. Robert Pianta, 
Dean of the Curry School of Education at the University of Virginia. Dr. Pianta spoke about the 
importance of using clear concise language and exemplars to facilitate the dialogue between 
principals and teachers. He also addressed the use of teacher evaluation as a catalyst for 
professional development. 

In 2014 the Principal Fellows attended a leadership institute hosted by Jefferson County Public 
Schools in Louisville, Kentucky. Jefferson County Public Schools was chosen because of the 
district being one of the first large districts to adopt and implement the Common Core State 
Standards. Through interaction and dialogue with key district and school leaders of Jefferson 
County, the Principal Fellows gained knowledge of implementation successes and challenges, 
assessment data, and valuable resources to support implementation efforts in Miami- Dade 
schools. 
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In 2015, Principal Fellows attended a leadership institute in New York City that also focused on 
the implementation of Common Core Standards. As a part of the institute, principals and other 
district administrators visited NYC schools located in Brooklyn, East Harlem, and Queens to 
observe and discuss strategies being implemented by teachers and administrators around 
implementation of the standards. 

Other Principal Supports 

The FMTI team provided other school supports that help principals share leadership and engage 
in data-driven decision making. One such support has been the administration and analysis of 
two schoolwide surveys—the School Culture Survey and the Instructional Practice Inventory— 
that enable data-driven decision making about instruction and schoolwide practices. The IPI is an 
observational assessment of instructional practices that measures the level of student engagement in 
learning. The School Culture Survey measures six factors of school culture—collaborative 
leadership, teacher collaboration, professional development, unity of purpose, trusting 
relationships between teachers, and learning partnership—and an efficacy factor. 

In 2011-2012, most schools participated in the surveys, but fewer did so in 2012-13 
(Exhibit 2-3). In particular, many fewer schools implemented the IPI survey. 


Exhibit 2-3. Participation in Surveys 

(n = 20) 



Percentage 
of schools 

2011-12 

Percentage 
of schools 

2012-13 

Instructional Practices Survey 

85% 

50% 

School Culture Survey 

90% 

80% 


The UF team decided to discontinue doing the two surveys for the 2013-14 year because of the 
relatively low level of interest from and use of data by principals given the high level of effort 
and resources required to conduct the surveys. However, the UF team planned to support the few 
schools that did want to continue to use the IPI survey to track efforts to improve student 
engagement. 

Fidelity of Principal Fellows Program 

Participation in the PPFC decreased each year even though there were fewer meetings. In Year 
1, almost two-thirds of the principals attended all the PPFC meetings (Exhibit 2-4). In Year 2, 
none of the principals attended all the meetings and most attended two or fewer of the four 
meetings. In Year 3, a growing number of principals attended only one meeting. 
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Exhibit 2-4. Attendance at Principal Fellows Meetings 

(n = 20) 


Number of 
Principal 

PLCs Attended 

Percentage 
of Principals 
2011-12 

Percentage 
of Principals 
2012-13 

Percentage 
of Principals 
2013-14 

0 

15% 

20% 

20% 

1 

10% 

5% 

35% 

2 

0% 

55% 

30% 

3 

0% 

20% 

15% 

4 

10% 

0% 

NA 

5 

65% 

NA 

NA 


In 2011-12, most of the principals attended both statewide Principal Leadership Institutes, and 
only 10% missed both institutes (Exhibit 2-5). In 2012-13, a quarter of the principals were not 
able to attend the one institute held, but the majority (75%) of principals were in attendance. In 
2013-14, only half the principals were able to attend the Principal Leadership Institute. 


Exhibit 2-5. Attendance at Principal Institutes 

(n = 20) 


Number of 
Principal 
Institutes 
Attended 

Percentage 
of Principals 
2011-12 

Percentage 
of Principals 
2012-13 

Percentage 
of Principals 
2013-14 

0 

10% 

25% 

50% 

1 

20% 

75% 

50% 

2 

70% 

NA 

NA 


In summary, implementation of the Principal Fellows program achieved medium or high fidelity 
in 80% of the intervention schools in Year 1 but was unable to reach a sufficient level of fidelity 
in enough schools in Years 2 and 3. In Year 3, only 40% of schools met medium or high fidelity 
on the Principal Fellows Program fidelity measure. 

Program Strengths 

Almost all principals who attended the PPLC meetings reported enjoying the new format of 
“principal talk” and felt it helped them develop a trusted professional network of principals who 
support each other, provide ideas for how to address issues, and share resources. For example, 
two principals shared the following: 

One of the things I did like about the format from last year was they were 
based in a school and the person that was hosting talked about a problem 
they were having at their school or something that we could all help with, 
bringing theory to practice, and worked on strategies that could assist that 
school. They shared good and bad, also what we could learn from them. 

I developed more collegial friendships, which is nice because we are a lot 
ofprincipals in the district, and we have something in common.... It’s kind 
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of nice because it's a little group that takes care of each other. We have 
that affinity, that friendship. So you feel safe. 

Your day can be full of negativity [as a principal]. You go to a PEC and 
you have all these things swirling around in your brain, you can vent and 
get advice, get solutions from other principals.... Everyone pipes in and 
you hear lots of voices, and it’s like you did a month’s worth of research 
and now you ’re ready to write your paper. I always say, a lot of the time 
you may be frustrated that you have to come to this morning meeting, but 
Eve never seen it take place with the principals that attend that they don 7 
always leave smiling or saying that it was well worth their time. 

Principals mentioned they also enjoyed networking with principals from other regions and that 
they do not normally get that chance to talk with colleagues outside of their regions. In 
particular, principals reported that getting outside the Miami-Dade County Public School district 
and their buildings helped them develop a deeper sense of camaraderie with each other and made 
them more open to thinking about new strategies. 

Program Challenges 

The biggest challenge for the Principal Fellows program was a lack of consistent participation by 
principals. Some of the reasons for decreased participation mentioned by the UF team and 
principals included the initial negative reaction to inquiry projects required in Year 1, the 
designation of several schools as ETO schools, which required them to attend to other pressing 
tasks and meetings, and the lack of a second statewide principal institute, which may have 
lowered the sense of community among the 20 FMTI school principals. 

One principal mentioned conflicts with district requirements leading to his low participation: 

The time and timelines [of the PPLC] overlap with other requirements 
that sometimes interfere with what any person can get to humanly. But I 
think the portions of the program that I have been involved in have been 
good. 

The FMTI implementation team decreased the number of PPLC meetings to try and make it 
easier for principals to attend, but that change did not increase participation levels and may have 
resulted in other unintended effects. For example, one principal reflected on how the change to 
fewer meetings may have reduced her ability to participate and get the full benefit of the 
program. 

I enjoyed those meetings but I remember I couldn 7 get to one, and then I 
felt like I only had two chances to see everyone.... I was probably one of 
the ones complaining about how it was a lot. But it almost forced you to 
do something, which was better than having the opportunity not to do 
something. So for me, in hindsight I would probably prefer to go back to 
meeting more. 

Finally, one principal who was interviewed did not feel that some of the PPLC meetings were 
relevant for improving practices at his school. He would have preferred a greater focus on 
analysis of school data and strategies to improve instruction. 


28 


Summer Leadership Institutes 

Another way FMTI supports more effective school leadership is through an annual multiday 
Summer Leadership Institute that generates school-specific action plans for the upcoming school 
year. Each school sent a leadership team, typically composed of the principal, assistant principal, 
teacher leader (who may be a teacher in the ECTLSI graduate degree program), and school- 
community liaison (in schools where such a position exists). These teams examined a variety of 
data on school culture, instructional practices, and student achievement to develop school 
improvement plans that they take back to their full faculties. Participants also used the summer 
institutes as a forum to learn about the practices and experiences of other schools in addressing 
such topics as using student data, protocols for PLC meetings, discipline, and improving school 
culture. 

The 2013 Summer Institute was held in late July. It focused on Appreciative Leadership, a 
method that identifies the key strengths of an organization and uses them as levers for change. 
Each school team was asked to define “a big audacious statement” about what they want their 
school to be, identify their strengths that could be used as levers for change, and narrow their hst 
down to 3 specific strategies. Teams built structures that represented their dreams and visions 
that labeled all the parts clearly so others could easily understand them. They then used big 
boards to lay out structured plans for how they will get to their dreams and visions. Teams took 
both of these items back to their schools to share with the rest of their faculty. 

Fidelity of the Summer Leadership Institute 

The intent is to have each school team include the principal, assistant principal, and teacher 
leaders. Schools differ on their team composition in large part because there is never a perfect 
time during the summer when everyone can attend. The UF team held its first and second 
Summer Leadership Institutes in June, but changes in principal assignments were made in July 
after the institutes. As a result, new principals did not have an opportunity to be part of the 
institute. In response, in 2013-14 school year, the team held the Summer Leadership Institute in 
late July, but several principals were on vacation at that time. 

Although most schools sent representatives to the Summer Leadership Institute (95% of schools 
in 2013, 85% of schools in 2012, and 95% of schools in 2011), most principals and more than 
half of the assistant principals did not attend in any given year and participation of school 
administrators dropped in Year 3 (Exhibit 2-6). However, almost all the schools had teacher 
leaders present, and more than half included a teacher in the FMTI ECTLSI graduate degree 
program in July 2013. Also, all the schools present at the institute submitted a school action plan. 
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Exhibit 2-6. Participation at Summer Leadership Institute 

(n = 20 schools) 


Type of Attendee 

2011-12 

N 

2011-12 

% 

2012-13 

N 

2012-13 

% 

2013-14 

N 

2013-14 

% 

Principal 

8 

40% 

8 

40% 

5 

25% 

Assistant principal 

6 

30% 

9 

45% 

7 

35% 

Teacher leader or community 
involvement specialist 

18 

90% 

17 

85% 

19 

95% 

FMTI ECTLSI graduate program 
student 

7 

35% 

7 

35% 

11 

55% 

No representatives 

1 

5% 

3 

15% 

1 

5% 

Submitted school action plan 

19 

95% 

17 

85% 

19 

95% 


Program Strengths 

Principals and teachers reported being excited about the plans developed during Summer 
Leadership Institutes. Principals reported the positive hopes they had for these plans and the 
progress being made on them. 

Teachers came up with an amazing plan, and they are doing it. It’s around 
building a positive school culture. There was a lot of negative sentiment 
about becoming an ETO school. Over the course of 3 days they dipped 
into that and unpacked it. They said let’s take the temperature and do a 
survey at the beginning of the year and come up with different committees 
to address aspects of school culture. The principal let them do it. I saw her 
last month and she is very excited about how the plan is going. 

At the Summer Institute we developed our focus on unity, and we felt good 
about our best practices. They gave us a big area to do protocols to build 
team and unity. We did a few of them at faculty meetings. Then class size 
hit and it all went down the tubes. The TLSI students are helping me get 
that moving again. 

Our motto is “building a stairway to success ” [shows a faculty meeting 
agenda that has five I’s: inspiration, illumination, inclusion, integrity, 
in quay]. My leadership team came up with that. It’s important enough to 
me that I want to remind people every time we step into a meeting that I 
want this to be our driving force. We did a protocol for the five I’s, what 
do they each mean, as part of our opening school debrief with whole 
faculty. 

It was useful work because we geared it towards what we wanted to do in 
school, 16 habits of mind, and how that tied into our ESAC, our PTA, and 
getting everyone involved, so it was something that we were able to work 
on and use as opposed to just doing it for the sake of doing it. 

Some teachers who attended the Summer Leadership Institute reported the meeting was an 
opportunity to talk about challenges with their school’s administration and across schools. 
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It was wonderful. I loved everything about it. You 're with people who are 
passionate about the same things that you are concerned about ...and you 
are sitting in a room with them and you are hearing powerful stuff, and 
they are having the same issues that you are having, the same concerns, 
maybe the same strengths, and you are there to share everything. 

Collaboration was the most useful, the ability to collaborate and listen to 
how other schools are doing things. We don 7 get a chance to do that. 

It was nice to see and hear the different points of view from the different 
levels. Sometimes they were the same as ours and sometimes they weren 7. 

Opening up lines of communication breaks down the walls. 

Program Challenges 

Despite high hopes, there were mixed reports from teachers about whether the school action 
plans developed at the summer institute were actually implemented. Teachers at a few schools 
reported the plan being successfully implemented. However, most of the teachers interviewed 
who attended the summer institute reported that it was hard to implement their plans back in their 
schools because the plans were not fully developed. For example, as shared by two teachers: 

I think we wanted more time to think about what they were going to do 
with our mission. And that is what made it slow to implement when you 
got back. Your opening school and thinking about another project is 
impossible unless you have a mind like Einstein’s. 

Having to implement something you haven 7 fully fleshed out and opening 
school, no one is going to do that. 

Participation of principals was also a challenge. Many of the principals interviewed did not 
participate in the 2013-14 Summer Leadership Institute or only attended one day because of 
conflicts with vacations or other commitments or because they felt it was an opportunity to 
delegate leadership to their assistant principals or teacher leaders. District leaders, the UF team, 
and teachers agreed that the Summer Leadership Institute worked best when the principal 
attended and was invested. They also noted it was a good way for new principals to begin work 
with their staff. When principals did not attend, teachers developed a school improvement plan 
that the principal may not buy into. To ameliorate this situation, when a principal did not attend 
the Summer Leadership Institute, the UF team met with him or her at the beginning of the school 
year to review with them the school plan that was developed and provided dates and other 
information about upcoming FMTI activities for the year. 

A few teachers reported that without representation from the school administration or broad 
representation from teachers across grades, it was difficult to develop meaningful school plans 
and the institute was much less valuable. 

Unfortunately, for many reasons, I only had another teacher with me. So 
really, that whole plan that we did was not effective because the staff who 
we needed there wasn 7. At least one administrator should have been there 
and at least one person from each grade level. 

Summer Institutes could be valuable if you go with principals and the 
leadership team. The year I went it was with the reading coach... We 


31 


wrote a plan during the training, but 1 just don't remember it being 
applied. 

Program Enhancements 

Because of some cost savings due to low enrollment in the ECTLSI graduate degree program, 
FMTI had surplus resources for the initiative. Based on input from teachers, principals, and the 
district, the FTMI program staff decided to apply these extra funds to three program 
enhancements that further promote the goals of the initiative: 

• The Assistant Principal Fellows program, which builds the leadership skills and 
professional support network of assistant principals across the 20 FMTI schools through 
Assistant Principal Professional Fearning Community (APPFC) meetings and an 
Assistant Principal Institute. 

• The Transition to Kindergarten Professional Fearning Community, which supports pre¬ 
kindergarten and kindergarten teachers in aligning curriculum between the two grades 
and smoothing the transition for children between preschool and kindergarten. 

• The post-baccalaureate (post-bac) program, which offers a non-degree bearing, four- 
course version of the ECTFSI program designed to help teachers who have participated 
in the Teacher Fellows program further improve their instructional practice. 

Assistant Principal Fellows 

The Assistant Principal Fellows program was introduced during the 2012-13 school year to 
address the needs of administrators holding this unique position in schools. APs’ roles and 
responsibilities are wide ranging - covering everything from curriculum to discipline to day-to- 
day management. Despite the importance of APs in the successful functioning of schools, district 
staff explained that APs are often overlooked because most professional development programs 
target either principals or teachers. As one district official described, 

Assistant principals at the elementary level are often forgotten in terms of 
everything. Communications, workshops, trainings, information sessions 
are all [targeted to] principals. The assistant principal and the principal 
really need to work as a unit in schools because a lot of times the 
principal isn 7 going to be the executive director. The day-to-day 
operations of schools often fall to assistant principals, who are at times 
disconnected from the information and the why. 

The idea for creating a fellows program to provide APs with the space and support to develop 
their leadership skills reportedly evolved organically through feedback program staff received 
from both principals and APs. The overall input was that APs would benefit from a similar 
program to the Principal Fellows program that supports networking and skills development. The 
AP fellows program has two components: APPFC meetings and an AP Institute. 

Assistant Principal Fellows meetings. In 2012-13, the APPFC met four times. The all-day 
meetings were held at varying locations throughout the district. Topics included appreciative 
leadership, reflective discourse, understanding the significance of learning communities, and 
identifying strategies for eliminating barriers in their work, among others. Participants also were 
encouraged to network and to build collaboration among assistant principals within and across 
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regions. Both district staff and UF faculty were extremely enthusiastic about the addition of the 
APPLCs. For example, one of the professors-in-residence explained, 

They are the best group we’ve worked with. They are so excited. If you 
know anything about school administrators, the AP is usually the gofer, 
they fill in the gaps. At the elementary level they do everything from 
curriculum to discipline. Unlike in high schools, they are usually the only 
AP in the building. In most cases principals don’t treat APs very 
respectfully and teachers blame them for everything..., They are very 
isolated. Having a chance to come together with other APs, not in a 
district meeting where they are just sitting and listening, but in a meeting 
where they actually get to speak and talk about challenges and support 
each other—it becomes a support group for these people... They are so 
hungry for the support of each other. 

In 2012-13, the APPLCs were open to APs from FMTI schools as well as from schools 
participating in a related initiative, Miami Counts. Among the 25 APs from FMTI schools 
(several schools employ more than one AP), eight or nine reportedly attended the APPLCs 
regularly. Low participation rates from FMTI schools may have been due to the nature of the 
APs’ job. One of the professors-in-residence explained that when APs did not attend it was 
usually because their principal said “no you can’t attend because I need you in the building and 
you can’t leave, or because they don’t know anyone in that group who gives them a push saying, 
‘hey come join us, it’s really great.’” 

Assistant Principal Institute. The first Assistant Principal Institute was held in October 2013 in 
Gainesville, Florida. The two-day AP institute provided an opportunity for APs to get away from 
their daily work to focus on learning and sharing practices with APs from other schools. The 
institute was open to all APs from the FMTI treatment schools but only seven attended. The 
institute focused on responsive leadership, utilizing the 5 Dimensions of Teaching and Learning 
Framework developed by the Center for Education Leadership at the University of Washington. 
The group also learned about the importance of early childhood as a foundation for later school 
success from UF faculty member and early childhood expert, Dr. Patricia Snyder. Finally, the 
APs had the opportunity to visit a local elementary school with a well-developed early childhood 
program, positive school culture, and a strong leadership team. In 2014, the APs traveled to 
Louisville, Kentucky for the second annual AP i3 Residential Institute, focused on the 
implementation of Common Core standards. 

Transition to Kindergarten PLC 

The second FMTI program enhancement is the Transition to Kindergarten PLCs, being 
implemented at all 20 FMTI schools. Introduced in the 2013-14 school year, the structure of the 
Transition PLCs replicates that of the traditional Teacher Fellows PLCs—they are inquiry based 
and operate at the school level. Unlike the Teacher Fellows however, Transition PLCs are only 
open to prekindergarten and kindergarten teachers (including paraprofessionals) and they are co¬ 
facilitated by one kindergarten and one prekindergarten teacher. 

Because of their specific emphasis on improving children’s transition between pre-school and 
kindergarten, FMTI district and program staff decided to give the Transition PLC facilitators 
more structure than Teacher Fellows facilitators by guiding their focus and giving them a list of 
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three inquiry topics to choose from: social emotional development, aligning standards, or barriers 
to transition. These topics were chosen because they are grounded in the early childhood 
literature. As the UF faculty member in charge of content for the Transitions PLCs explained, 

When we did the facilitator training we did some foundational reading on 
why this is important and why we are concerned, and then we have ready- 
to-go, stockpiled readings for whichever topic they pick to support them. 

In general, UF faculty members view the Transition PLCs as an opportunity to place added focus 
on the early childhood core of the FMTI: 

It’s something we’ve really wanted to address to a greater extent in the 
graduate program, but it’s been difficult because we don’t have that many 
Pre-K or kindergarten teachers. We do talk about it in the Policy and 
Transitions course, but it’s pitched more as transition between grade 
levels is always a big change for a child and is always something we 
should be aware of. But the kind of transition to school work that the 
district really wanted, we haven 7 been able to do because we just haven 7 
had Pre-K and kindergarten teachers in large enough numbers. So I think 
this allows us to address what has very much been a district concern but 
has been difficult for us to hit in other places. 

At the time we interviewed school staff, the teacher facilitators for the Transition PLCs had 
attended training sessions but the PLCs had not yet commenced meeting. Although the 
implementation of the Transition PLCs was only beginning to unfold, there were positive 
expectations for the program from both district staff and teachers. 

Post-Baccalaureate Professional Development 

The third FMTI program enhancement is a non-degree bearing, post-baccalaureate (post-bac) 
program, consisting of four of the core ECTLSI courses. The courses included in the post-bac 
program are Guided Teacher Inquiry, Culturally Responsive Classroom Management, 
Differentiated Instruction, and Families. These four courses were selected because of their 
emphasis on improving teachers’ instructional practice. As one UF faculty member explained, 

We picked courses that were targeted towards improving instruction. They 
were not supposed to be theoretical courses, they were supposed to be 
practical, applied courses that will help them improve practice. I think the 
one exception to that is the inquiry course, because that is really an 
approach, or a stance towards studying your own practice. We felt that 
one was critical, because it helps them develop the idea of focusing on 
their own practice, to help them improve it... So the purpose of the post- 
bac was really to only focus on the master teacher piece of our three 
goals. 

The logic behind the creation of the post-bac program was to provide an additional avenue of 
professional development for teachers that required less of a commitment than the full ECTLSI 
graduate program, did not require taking the GRE, and was not restricted to teachers in early 
grades (preschool through third grade). Like the ECTLSI graduate program, the post-bac 
graduate courses were free to participating teachers. 
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Because the post-bac course sequence does not lead to a degree, however, participation does not 
result in salary increases. The incentives to teachers are receiving four graduate courses for free 
(which is one third of a degree program), trying out a graduate program to see if it’s plausible for 
them, and completing coursework toward recertification (although it is more than is needed). 

Despite these benefits, both recruitment and retention were disappointing. Although FMTI had 
sufficient funding for 40 post-bac teachers, only 17 teachers enrolled. Of those initial 17, five 
teachers dropped before courses began and one more dropped after the first couple of classes. 
Consequently, at the time of our data collection, only eleven teachers remained in the post-bac 
program. When teachers who were not in the post-bac program were asked why they did not 
apply to the program, most teachers cited their busy schedules or conflicts with their families or 
personal lives. It is possible that, given how busy teachers are already, an incentive greater than 
free graduate coursework is necessary to motivate them to add to their already taxing schedules. 

Regarding retention, several program staff speculated that in their efforts to reduce the barriers to 
recruitment, they may have unwittingly created a situation where enrolling was so easy that 
teachers did not form any attachment to the post-bac program. 

Our speculation is that people don’t feel they have skin in the game if they 
are not paying for coursework. The entry bar was set so low, they didn ’t 
have to take the GRE, no application fee. They are essentially in without 
any attachment that keeps them there or anything on their part. 

The whole program is a freebie to teachers, but if they have to take the 
GRE and they know if they hang in there they’ll have a graduate degree 
then perhaps there will be a greater degree of commitment of those 
teachers. If you ’re handed it and didn’t have to do anything, then for some 
teachers it may not be the same degree of commitment and they don’t see 
a product. 

Chapter Summary 

FMTI is a complex and multifaceted initiative. It is comprised of multiple programmatic 
components, each targeted at different audiences; it is operating in 20 different schools of 
varying size, demographics, resources, and needs. Still, FMTI partners were successful with 
offering high quality programs for principals and teachers. All of the courses for the ECTLSI 
graduate degree program were developed, implemented, and well received. The Teacher Fellows 
program was successfully implemented in all the treatment schools. The UF team held statewide 
institutes and local meetings for principals to learn about and observe new leadership practices 
and develop a professional network and learning community with other principals of FMTI 
schools. However, recruitment and participation levels in the ECTLSI graduate degree program 
and Principal Fellows program was a challenge and resulted in the program not achieving the 
level of implementation fidelity needed to accurately assess the model’s impact. Further, 
requirements related to schools being designated high priority schools by the Educational 
Transformation Office made implementing some of the instructional practices learned through 
the graduate program a challenge. FMTI partners responded to these challenges by incorporating 
new program enhancements and working on the scheduling of program activities to avoid 
conflicts with other district activities. 
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3. Impact Analysis: Teachers 


FMTI is designed to support the development of a professional learning community among 
school staff and improve teachers’ skills in instruction, research, and leadership. Ultimately these 
improvements to school culture, teacher leadership, and classroom practice are hypothesized to 
lead to increased student achievement, stronger emotional and social foundations for student 
learning, and greater student engagement. This chapter focuses on the short-term teacher and 
school outcomes. We present the results from two major data collection efforts: (1) pre- and 
post-treatment surveys of all teachers in treatment and control schools, and (2) observations of 
classroom instruction for teachers in the job-embedded graduate program and matched 
comparison teachers in control schools. 

Teacher Survey 

As described in Chapter 1, the teacher survey was designed to capture teachers’ reports on 
measures related to school culture (i.e., collaboration within the school, trusting relationships 
between teachers, principal leadership), teacher leadership (i.e., involvement in leadership roles, 
governance activities, and outreach activities) and teacher instructional practices (i.e., learner- 
centered instruction, assessment-informed practice, regular use of a variety of assessments, 
developmentally-appropriate instruction, higher-order thinking skills, differentiated instruction, 
culturally-responsive instruction, family partnerships, early childhood knowledge, and general 
teaching knowledge). 

We begin by presenting the confirmatory teacher survey results for both the RCT schoolwide 
analysis (all teachers in treatment schools compared with all teachers in control schools) as well 
as for the embedded QED that focuses only on treatment teachers who participated in the 
ECTLSI graduate program (compared with all Pre-K-3 teachers in control schools). For ease of 
comparison, we present the schoolwide and ECTLSI teacher results side by side for each 
outcome measure. Next, we present results from an exploratory analysis that restricts the 
treatment group sample to only teachers in schools that attained medium to high fidelity to the 
FMTI model. Appendix A provides more detail on the data, sample, baseline equivalence, and 
analytic approach for each analysis. 

Confirmatory Results 

For each outcome, we present the model-adjusted means for treatment (FMTI participants) and 
control groups. Means presented are adjusted to account for differences in baseline means and 
teacher demographics that may have existed between treatment and control schools. We found 
no statistically significant differences between teacher reports on school culture and professional 
learning community for either the schoolwide or ECTLSI teacher analyses. For self-reported 
involvement in teacher leadership activities, we found a consistent, positive impact of the FMTI 
program on ECTLSI teachers, but not on the school as a whole. With regards to classroom 
practices, we found no significant differences for the majority of outcomes, however we did find 
that the FMTI program had a small negative impact on schoolwide teacher reports of 
differentiating instruction and a positive impact on both the early childhood knowledge and 
general teaching knowledge of ECTLSI teachers. 

Collaboration around instruction. Teachers, on average, reported collaborating with 
colleagues around instruction between once or twice a month and once or twice a week 
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(Exhibit 3-1). There was no statistically significant difference between teachers in the treatment 
and control schools in their report of collaboration during the 2013-14 school year for either the 
schoolwide or ECTLSI teacher analyses. 

Exhibit 3-1. Frequency of Collaboration around Instruction, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 

Last year, with other teachers I... 

• Discussed what I/they learned at a workshop or 
conference. 

• Shared, discussed, and analyzed student work. 

• Discussed and analyzed instructional issues and problems. 

• Shared and discussed research on effective teaching 
methods. 

• Developed teaching materials or activities for particular 
classes. 

• Discussed student assessment data to make decisions about 
instruction. 


Scale Points: 

1 = Never 

2 = A few times a year 

3 = Once or twice a month 

4 = Once or twice a week 

5 = Almost every day 
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Trusting relationships between teachers. Teachers, on average, agreed to strongly agreed with 
positive statements about trusting relationships between teachers (Exhibit 3-2). There was no 
statistically significant difference between teachers in the treatment and control schools on their 
perceptions of teacher collegiality during the 2013-14 school year for either the schoolwide or 
ECTLSI teacher analyses. 

Exhibit 3-2. Trusting Relationships between Teachers, 2013-14 School Year 


Schoolwide 


ECTLSI Teachers 



Strongly Disagree 


Strongly Agree 


Survey Items: 


Scale Points: 

Last year, teachers in this school generally... 


1 = Strongly disagree 

• Felt supported by colleagues to try out new ideas. 


2 = Disagree 

• Trusted each other. 

• Felt responsible to help each other do their best. 


3 = Agree 

• Felt comfortable sharing their challenges with each other. 


4 = Strongly agree 

• Were open to advice and feedback trom their peers. 
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Effective PreK-3 principal leadership. Teachers, on average, agreed to strongly agreed with 
positive statements describing their principal’s quality of leadership (Exhibit 3-3). There was no 
statistically significant difference between teachers in the treatment and control schools on their 
evaluation of principal leadership during the 2013-14 school year for either the schoolwide or 
ECTLSI teacher analyses. 

Exhibit 3-3. Principal Leadership, 2013-14 School Year 


Schoolwide 


ECTLSI Teachers 



Strongly Disagree 


Strongly Agree 


Survey Items: 

Last year, the principal at this school... 

• Set high standards for teaching. 

• Communicated a commitment to high-quality PreK-3 
learning. 

• Actively monitored the quality of teaching in this school. 

• Ensured that teachers had dedicated time for collaboration. 

• Encouraged school staff to take on leadership roles. 

• Supported successful transitions of students from one grade 
to the next. 


Scale Points: 

1 = Strongly disagree 

2 = Disagree 

3 = Agree 

4 = Strongly agree 
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Teacher leadership. The level of involvement of teachers in teacher leadership averaged around 
a few times a year (Exhibit 3-4). On average, ECTLSI teachers reported slightly higher levels of 
involvement in leadership roles than similar teachers in control schools during the 2013-14 
school year. However, there was no statistically significant schoolwide difference in involvement 
in teacher leadership between teachers in treatment and control schools. 

Exhibit 3-4. Teacher Leadership, 2013-14 School Year 



Never 


Frequency 


Once a week 
or more 


*p < .05 


Survey Items: 

Last year, I... 

• Developed curriculum to be used by a team of teachers. 

• Assisted in the design or planning of staff development 
activities. 

• Led staff development activities. 

• Facilitated text-based study groups. 

• Participated in peer observation, coaching, or modeling 
activities. 

• Facilitated teacher meetings (e.g., grade level, faculty 
meetings, professional learning communities). 


Scale Points: 

1 = Never 

2 = A few times a year 

3 = Once a month 

4 = 2 or 3 times a month 

5 = Once a week or more 
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Governance activities. On average, ECTLSI teachers were more likely to engage in a 
governance activity during the 2013-14 school year than similar teachers in control schools 
(Exhibit 3-5). Schoolwide, teachers in treatment schools were also more likely to report engaging 
in a governance activity than teachers in control schools. 

Exhibit 3-5. Governance Activities, 2013-14 School Year 


ECTLSI 

Teachers 


Schoolwide 



*p < .05 


Survey Items: 


This measure is an indicator 

Last year, I... 


where Outreach = 1 if a 
teacher indicated they 

• Held a designated leadership role in the school. 


participated in any of the three 

• Helped decide how discretionary school funds should be 


activities listed, and otherwise 

used. 


Outreach = 0. 

• Helped develop the school improvement plan. 
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Early learning outreach activities. On average, ECTLSI teachers were more likely to 
participate in an early learning outreach activity during the 2013-14 school year than similar 
teachers in control schools (Exhibit 3-6). By contrast, there was no statistically significant 
schoolwide difference in involvement in governance activities between teachers in treatment and 
control schools. 


Exhibit 3-6. Outreach Activities, 2013-14 School Year 



*p < .05 


Survey Items: 

Last year, I... 

• Worked with preschool programs or family care centers in 
my community to promote school readiness activities. 

• Promoted linkages between feeder early learning programs 
and my elementary school. 

• Advocated for early learning programs and my elementary 
school. 

• Spoke at school board meetings to advocate for policies, 
programs, or funding that promote child well-being. 


This measure is an indicator 
where Outreach = 1 if a 
teacher indicated they 
participated in any of the 
activities listed. 
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Learner-centered instruction. Teachers, on average, reported using learner-centered 
instructional strategies with students about once or twice a week (Exhibit 3-7). There was no 
statistically significant difference between teachers in the treatment and control schools in their 
frequency of engaging in learner-centered instruction during the 2013-14 school year for either 
the schoolwide or ECTLSI teacher analyses. 

Exhibit 3-7. Learner-Centered Instruction, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 


Scale Points: 

Last year, I asked my students to... 


1 = Never 

• Make choices about their own activities (what they did 


2 = A few times a year 

and/or how they did it). 

• Discuss their ideas and learning with other students in 


3 = Once or twice a month 

formal groupings. 


4 = Once or twice a week 

• Participate in class meetings or discussions to share 
feelings, solve problems together, or talk about personal 


5 = Almost every day 

interests. 
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Assessment-informed practice. Teachers, on average, reported using data to inform practice 
between once or twice a week and almost every day (Exhibit 3-8). There was no statistically 
significant difference between teachers in the treatment and control schools in their frequency of 
engaging in learner-centered instruction during the 2013-14 school year for either the schoolwide 
or ECTLSI teacher analyses. 

Exhibit 3-8. Assessment-Informed Practice, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 


Scale Points: 

Last year, 1 used assessment data to... 


1 = Never 

• Individualize instruction for each student. 


2 = A few times a year 

• Inform curricular and lesson planning. 


3 = Once or twice a month 

• Evaluate the overall effectiveness of my instructional 



practice. 


4 = Once or twice a week 



5 = Almost every day 


44 






Regular use of a variety of assessments. Teachers, on average, reported using between three or 
four (of a total of four) different types of assessment at least once during the 2013-14 school year 
(Exhibit 3-9). There was no statistically significant difference between teachers in the treatment 
and control schools on the number of assessment activities used for either the schoolwide or 
ECTLSI teacher analyses. 

Exhibit 3-9. Regular Use of a Variety of Assessments, 2013-14 School Year 


Schoolwide 


ECTLSI Teachers 



Number of activities done at least once 


Survey Items: 

Last year, I used student assessment results to... 

• Direct observation looking for specific skills 

• Direct assessment or testing (e.g., district tests or chapter 
tests) 

• Ongoing formative assessment (progress monitoring) 

• Portfolios of students’ work samples 


This measure is constructed by 
counting the number of 
different activities a teacher did 
at least a few times a year. 
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Developmentally appropriate practices. Teachers, on average, reported using developmentally 
appropriate practices about once or twice a week (Exhibit 3-10). There was no statistically 
significant difference between teachers in the treatment and control schools in their frequency of 
engaging in developmentally appropriate practices during the 2013-14 school year for either the 
schoolwide or ECTLSI teacher analyses. 

Exhibit 3-10. Developmentally Appropriate Practices, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 


Scale Points: 

Last year, I asked my students to... 


1 = Never 

• Use manipulatives, real objects (e.g., plants, animals), and 


2 = A few times a year 

concrete materials as part of their learning experiences. 

• Engage in inquiry through experiments or projects. 


3 = Once or twice a month 

• Engage in open exploration or play. 


4 = Once or twice a week 

• Listen, sing, and/or move to music as part of my lessons. 

• Represent what they learn in ways other than writing (art. 


5 = Almost every day 

constructions, dramatizations). 
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Emphasis on higher-order thinking skills. Teachers, on average, asked students to engage in 
higher-order thinking skills between once or twice a week and almost every day (Exhibit 3-11). 
There was no statistically significant difference between teachers in the treatment and control 
schools in their frequency of engaging students in higher order thinking skills during the 2013-14 
school year for either the schoolwide or ECTLSI teacher analyses. 

Exhibit 3-11. Emphasis on Higher-Order Thinking Skills, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 

Last year, asked my students to... 

• Discuss possible solutions to problems with other students. 

• Think aloud as they tried to solve a problem. 

• Apply content knowledge to real-world scenarios. 

• Discuss their point of view about something they read or I 
read to them. 

• Discuss connections between a reading and real-life people 
or situations. 

• Generate a prediction or hypothesis. 

• Think about the factors that influenced an idea or caused an 
event to happen. 


Scale Points: 

1 = Never 

2 = A few times a year 

3 = Once or twice a month 

4 = Once or twice a week 

5 = Almost every day 
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Differentiated instruction. Teachers, on average, reported differentiating instruction between 
once or twice a week to almost every day (Exhibit 3-12). Schoolwide, teachers in treatment 
schools reported differentiating instruction slightly less than teachers in control schools during 
the 2013-14 school year. There was no statistically significant schoolwide difference in 
differentiating instruction between ECTLSI teachers and similar teachers in control schools. 

Exhibit 3-12. Differentiated Instruction, 2013-14 School Year 



Never 


Almost every 

Frequency day 


*p < .05 


Survey Items: 


Scale Points: 

Last year, I... 


1 = Never 

• Used a wide variety of instructional strategies that are 


2 = A few times a year 

related to different learning styles. 

• Provided multiple ways for students to demonstrate 


3 = Once or twice a month 

knowledge and skills. 


4 = Once or twice a week 

• Used pre-assessments of students’ skills to plan instruction. 

• Determined students’ interests to help connect learning to 


5 = Almost every day 

their specific interests. 
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Culturally responsive instruction. Teachers, on average, reported using culturally responsive 
instruction between once or twice a month and once or twice a week (Exhibit 3-13). There was 
no statistically significant difference between teachers in the treatment and control schools in 
their frequency of using culturally responsive instruction during the 2013-14 school year for 
either the schoolwide or ECTLSI teacher analyses. 

Exhibit 3-13. Culturally Responsive Instruction, 2013-14 School Year 


Schoolwide 


ECTLSI 

Teachers 



Never 


Almost every 

Frequency day 


Survey Items: 

In my class last year, I... 

• Adapted lessons from texts or curriculum guides to the 
cultural background of my students. 

• Provided opportunities for students to value and explore 
diversity (such as cultural heritage) in themselves and 
others. 

• Used instructional activities that build on home and family 
experiences. 

• Used classroom materials that reflect the backgrounds and 
experiences of my students (e.g., pictures of familiar 
places). 


Scale Points: 

1 = Never 

2 = A few times a year 

3 = Once or twice a month 

4 = Once or twice a week 

5 = Almost every day 
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Family partnerships. On average, teachers reached out to roughly 51-75% of families through 
multiple methods (Exhibit 3-14). There was no statistically significant difference between 
teachers in the treatment and control schools on the proportion of families to whom they reached 
out during the 2013-14 school year for either the schoolwide or ECTLSI teacher analyses. 

Exhibit 3-14. Family Partnerships, 2013-14 School Year 


Schoolwide 


ECTLSI Teachers 


4.10 

■° 4 ■ Treatment 

Control 

4.14 

- 1 - 1 

1 2 3 4 5 6 

0% 1-25% 26-50% 51-75% 76-99% 100% 

Percentage of Families Participating in Activities 




Survey Items: 


Scale Points: 

Last year, to reach out to parents, I... 


1 = 0% of families 

• Called or sent a personal note/email to discuss a concern. 


2 = 1-25% of families 

• Called or sent a personal note/email to share positive news. 


3 = 26-50% of families 

• Talked with them informally before or after class. 



• Sent home activities for them to do with their children to 


4 = 51-75% of families 

support student learning. 


5 = 76-99% of families 

• Used school-based resources such as a Community 



Involvement Specialist to reach out to families. 


6 = 100% of families 
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Early childhood instructional knowledge. Teachers, on average, reported close to moderate 
early childhood content knowledge (Exhibit 3-15). On average, ECTLSI teachers reported 
slightly higher levels of early childhood knowledge than similar teachers in control schools 
during the 2013-14 school year. There was no statistically significant schoolwide difference in 
self-reported early childhood knowledge between teachers in treatment and control schools. 

Exhibit 3-15. Early Childhood Knowledge, 2013-14 School Year 



No Extensive 

Knowledge Knowledge 


***p < .001 


Survey Items: 


Scale Points: 

My current level of knowledge of... 


1 = No knowledge 

• Early child development from birth to age 5. 


2 = Minimal knowledge 

• Child assessment. 


3 = Moderate knowledge 

• Strategies for promoting family engagement. 



• Florida Early Learning and Developmental Standards for 


4 = Extensive knowledge 

4-year-olds. 
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General instructional knowledge. Teachers, on average, reported moderate to extensive general 
instructional knowledge (Exhibit 3-16). ECTLSI teachers reported slightly higher levels of 
general instructional knowledge, on average, than similar teachers in control schools during the 
2013-14 school year. There was no statistically significant schoolwide difference in self-reported 
general instructional knowledge between teachers in treatment and control schools. 

Exhibit 3-16. General Instructional Knowledge, 2013-14 School Year 



No Extensive 

Knowledge Knowledge 


**p < .01 


Survey Items: 


Scale Points: 

My current level of knowledge of... 


1 = No knowledge 

• Strategies for using ongoing student assessment to plan 


2 = Minimal knowledge 

instruction. 


3 = Moderate knowledge 

• Strategies to create a positive learning environment. 



• Strategies for integrating curriculum. 


4 = Extensive knowledge 

• Florida Early Learning and Developmental Standards for 



4-year-olds 
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Exploratory Results 

In the section below, we present data from the exploratory analysis of the impact of FMTI on the 
subset of FMTI schools with a medium to high implementation fidelity rating. Here we present 
only the schoolwide analysis. We found no statistically significant differences between teachers 
in FMTI schools and those in control schools on the majority of outcomes, but found statistically 
significant positive findings for teachers in the medium and high fidelity FMTI schools on 
trusting relationships between teachers, teacher leadership, regular use of a variety of 
assessments, and family partnerships. In the interest of parsimony, we present results for 
significant findings only. Appendix C presents model estimates from all exploratory analyses. 

Trusting relationships between teachers. Teachers, on average, agreed to strongly agreed with 
positive statements about trusting relationships between teachers (Exhibit 3-17). Teachers in the 
medium/high fidelity treatment schools reported somewhat stronger teacher collegiality during 
the 2013-14 school year than teachers in the control schools. 

Exhibit 3-17. Trusting Relationships between Teachers, 2013-14 School Year, 

Medium/High Fidelity Schools 


■ Medium/high 
fidelity treatment* 

Control 

i-1-1-1 

12 3 4 

Strongly Strongly 

Disagree Agree 

*p < .05 



Survey Items: 


Scale Points: 

Last year, teachers in this school generally... 


1 = Strongly disagree 

• Felt supported by colleagues to try out new ideas. 


2 = Disagree 

• Trusted each other. 

• Felt responsible to help each other do their best. 


3 = Agree 

• Felt comfortable sharing their challenges with each other. 


4 = Strongly agree 

• Were open to advice and feedback from their peers. 
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Teacher leadership. The level of involvement of teachers in teacher leadership averaged around 
a few times a year (Exhibit 3-18). Teachers in the medium/high fidelity treatment schools 
reported slightly more involvement in teachers’ leadership activities during the 2013-14 school 
year than teachers in the control schools. 

Exhibit 3-18. Teacher Leadership, 2013-14 School Year, Medium/High Fidelity 

Schools 


■ Medium/high fidelity 
treatment* 

Control 


5 

Once a week 
or more 

*p < .05 


1 

Never 


Frequency 



Survey Items: 

Last year, I... 

• Developed curriculum to be used by a team of teachers. 

• Assisted in the design or planning of staff development 
activities. 

• Led staff development activities. 

• Facilitated text-based study groups. 

• Participated in peer observation, coaching, or modeling 
activities. 

• Facilitated teacher meetings (e.g., grade level, faculty 
meetings, professional learning communities). 


Scale Points: 

1 = Never 

2 = A few times a year 

3 = Once a month 

4 = 2 or 3 times a month 

5 = Once a week or more 
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Regular use of a variety of assessments. Teachers, on average, reported using between three or 
four (of a total of four) different types of assessments at least once during the 2013-14 school 
year (Exhibit 3-19). Teachers in the medium/high fidelity treatment schools reported using 
slightly more types of assessment than teachers in the control schools. 

Exhibit 3-19. Regular Use of a Variety of Assessments, 2013-14 School Year, 

Medium/High Fidelity Schools 


■ Medium/high 
fidelity treatment* 

Control 



3.73 


1 


1 


2 


3 


4 


Number of activities done at least once 


*p < .05 


Survey Items: 

Last year, I used student assessment results to... 

• Direct observation looking for specific skills 

• Direct assessment or testing (e.g., district tests or chapter 
tests) 

• Ongoing formative assessment (progress monitoring) 

• Portfolios of students’ work samples 


This measure is constructed by 
counting the number of 
different activities a teacher did 
at least a few times a year. 
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Family partnerships. On average, teachers reached out to approximately 51-75% of families 
during the 2013-14 school year through multiple methods (Exhibit 3-20). The teachers in 
medium/high fidelity treatment schools reached out to a slightly higher percentage of families 
than teachers in the control schools. 

Exhibit 3-20. Family Partnerships, 2013-14 School Year, Medium/High Fidelity 

Schools 


■ Medium/high fidelity 
treatment** 

Control 



1 2 3 4 5 


6 


0 % 1 - 25 % 26 - 50 % 51 - 75 % 76 - 99 % 100 % 

Percentage of Families Participating in Activities 


**p < .01 


Survey Items: 


Scale Points: 

Last year, to reach out to parents, I... 


1 = 0% of families 

• Called or sent a personal note/email to discuss a concern. 


2 = 1-25% of families 

• Called or sent a personal note/email to share positive news. 


3 = 26-50% of families 

• Talked with them informally before or after class. 



• Sent home activities for them to do with their children to 


4 = 51-75% of families 

support student learning. 


5 = 76-99% of families 

• Used school-based resources such as a Community 



Involvement Specialist to reach out to families. 


6 = 100% of families 
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Classroom Observations of ECTLSI Teachers and Matched Comparison Teachers 

The ECTLSI program aims to develop a cadre of teachers with deeper knowledge of the early 
learning foundations promoted by the FMTI, as well as strong research and leadership skills at 
each school. By design, these teachers received a much more intensive treatment than non- 
ECTLSI teachers in treatment schools and were consequently expected to make the strongest 
improvement to their instructional practice. To understand how the FMTI program impacted the 
classroom instruction of ECTLSI teachers, researchers conducted and scored classroom 
observations of ECTLSI teachers and a group of matched comparison teachers located within 
control schools at two points in time using the Classroom Assessment Scoring System (CLASS). 
A baseline observation occurred early in the teachers’ first year of the program and a follow-up 
observation occurred toward the end of the program. For more detail on the data collection, 
scoring, sample attrition, baseline equivalence, and analytic approach, please see Chapter 1 and 
Appendix A. 

Due to the differential attrition between treatment and control groups for the follow-up 
classroom observations, described in Chapter 1, it is possible that the results from this analysis 
may be biased. For example, if control teachers who agreed to be observed at follow-up are those 
who made more recent progress than teachers missing their follow-up observation, the control 
group may appear artificially positive on classroom instruction practice as compared to the 
treatment group. Although we used propensity score weighting to balance baseline observation 
scores with a standardized mean difference of greater than .25, this kind of selection bias on the 
outcome may not be adjusted away. 

Below we present a graph with the findings from the classroom observations (Exhibit 3-21). For 
each domain observed, we present the model-adjusted means for treatment (ECTLSI teachers) 
and control groups. The analysis did not find statistically significant differences between the 
ECTLSI teachers and their matched comparison teachers on emotional support or classroom 
organization domains. However, we did find a statistically significant positive impact of the 
FMTI program on the instructional support domain. The ECTLSI teachers were found to have 
performed 1.7 points better than the matched comparison teachers on the instructional support 
domain (Exhibit 3-21). 
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Exhibit 3-21. Mean CLASS Scores, for Master’s Teachers and Matched 
Comparisons, 2013-14 School Year 



■ ECTLSI Teachers Matched Comparison Teachers 


***p<. 001 


Chapter Summary 

The evaluation found that most schoolwide outcomes related to school culture and instructional 
practices did not differ between treatment schools and comparison schools as measured through 
a teacher survey, but there were a few positive and significant results for teachers in medium or 
high fidelity schools compared to teachers in comparison schools. Also, ECTLSI teachers— 
those receiving the most intensive professional development—outperformed their matched 
comparison teachers on the instructional support domain of the CLASS and more of the practices 
measured by the teacher survey. While not conclusive about the effectiveness of LMTI, these 
findings are promising. 
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4. Impact Analysis: Students 


The ultimate goal of the FMTI program is to improve student outcomes. This chapter presents 
results from analyses examining the impact of FMTI on student reading and math test scores. We 
present the results of the schoolwide analyses of all students in treatment schools compared with 
all students in control schools for the RCT, as well as results from the embedded QED that 
focuses only on students who were taught by treatment teachers who participated in the ECTLSI 
program. Please see Appendix A for detailed descriptions of the analyses. 

Schoolwide Impact on Student Achievement 

This section presents analyses of reading and math test scores of students in schools randomly 
assigned to treatment and control conditions in order to estimate the schoolwide impact of FMTI 
on student achievement. For these analyses, we included students who were enrolled in treatment 
and control schools in grades Pre-K through 2 in the spring of the baseline year (2010-11), right 
before random assignment took place. We examined their achievement outcomes three years 
later in 2013-14, when they were mostly in grades 2-5 . 

Confirmatory Results 

The confirmatory analysis estimates the impact of FMTI on all students in grade Pre-K to 2 at 
baseline. It is an intent-to-treat analysis that includes students who changed schools during the 
intervention period, which provides an unbiased estimate of the impact of schools being 
randomly assigned to the treatment versus control group on all students at baseline. 

For reading and math separately, we present the model-adjusted means for treatment students in 
FMTI schools at baseline and control students in control schools at baseline. We did not find 
statistically significant differences in student reading and math achievement between those in 
FMTI and control schools. Appendix C presents model estimates for confirmatory analyses. 


3 We included baseline Pre-K students who were retained and were in first grade in 2013-14. 
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Exhibit 4-1. Model-adjusted Outcome Scores for Schoolwide Impact Analysis 

Student Math and Reading, 2013-14 School Year 


RCT Student Reading 



RCT Student Math 



I-1-1- 

-0.3 -0.2 -0.1 0 


■ Treatment 
Control 


- 1 - 1 - 1 

0.1 0.2 0.3 


Exploratory Results 

The exploratory analyses investigated the impact of FMTI on all students in grades Pre-K to 2 at 
baseline who stayed in the same schools over the three years of intervention, and on those who 
also stayed in the same schools with medium/high fidelity of program implementation. These 
exploratory analyses provide insights into FMTI’s impact on students who received treatment for 
the maximum amount of time and on those who received prolonged treatment in schools that 
truly implemented the program. Neither analysis found statistically significant differences in 
student reading and math achievement between FMTI and comparison schools. Appendix C 
presents model estimates for all exploratory analyses. 

ECTLSI Teacher Impact 

The confirmatory analysis of ECTLSI teacher impact compared the reading and math test scores 
of first- to third-grade students between ECTLSI teachers and matched comparable teachers in 
control schools, adjusting for differences between non-ECTLSI teachers in treatment schools and 
non-matched teachers in control schools (i.e., a difference-in-differences analysis). This analysis 
examined whether there was an ECTLSI teacher impact on student outcomes on top of the FMTI 
schoolwide impact. We included in the analysis teachers who were linked to students with 
reading and math test scores in 2013-14, and who also had baseline survey data so that we could 
match ECTLSI and control-school teachers. We examined student achievement outcomes in 
2013-14, after students had received instruction from ECTLSI and comparison teachers for a 
year. 


Confirmatory Results 

We present the model-adjusted means for reading and math separately by the four groups of 
teachers in the difference-in-differences analysis: ECTLSI teachers, matched teachers in 
comparison schools, non-ECTLSI teachers in treatment schools, and nonmatched teachers in 
control schools. We did not find statistically significant differences between ECTLSI and 
matched teachers, adjusting for the differences between non-ECTLSI teachers in treatment 
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schools and non-matched teachers in control schools. Appendix C presents model estimates for 
from all confirmatory analyses. 

Exhibit 4-2. Model-adjusted Outcome Scores for ECTLSI Teacher Impact Analysis 


Student Reading, 2013-14 School Year 


ECTLSI teachers 


Non-ECTLSI teachers 
in treatment schools 

Matched teachers 
in control schools 

Nonmatched teachers 
in control schools 


-0.30 


-0.078 



0.30 


Student Math, 2013-14 School Year 


ECTLSI teachers 

Non-ECTLSI teachers 
in treatment schools 

Matched teachers 
in control schools 

Nonmatched teachers 
in control schools 


-0.300 -0.200 -0.100 



0.000 0.100 0.200 0.300 

Z Scores 


Chapter Summary 

There were no significant differences between students in treatment and control schools on math 
or reading achievement. These findings were consistent for all analyses conducted. 
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5. Conclusion 


Even though FMTI partners were successful at launching their four-pronged program, 
recruitment and participation levels in the ECTLSI graduate degree program and Principal 
Fellows program were a challenge and resulted in the program not achieving the level of 
implementation fidelity needed to accurately assess the model’s impact. Further, requirements 
related to schools being designated high priority schools by the Educational Transformation 
Office made implementing some of the instructional practices learned through the graduate 
program a challenge. 

With this less than optimal implementation of the program, most of the desired outcomes were 
not achieved. Most of the school culture, teaching quality, and instructional practices measured 
by the teacher survey did not change as a result of the program. There was growth in teacher 
participation in governance activities in FMTI treatment schools but a slight decline in their use 
of differentiated instruction. 

A more positive story emerged for FMTI treatment schools that achieved medium or high 
fidelity of implementation across the three years. Exploratory analyses found positive changes in 
trusting relationships between teachers, participation in leadership roles, family partnerships, and 
use of a greater variety of assessments in schools with medium or high fidelity of FMTI 
implementation compared to control schools. 

The evaluation also found some significant impacts on teachers who participated in the ECTFSI 
graduate program. Specifically, ECTFSI teachers had slightly higher levels of involvement in 
leadership roles, self-reported early childhood knowledge, self-reported general instructional 
knowledge, participation in governance activities, and engagement in outreach activities than 
similar teachers in control schools. The ECTFSI teachers also improved the quality of their 
instructional support compared to matched teachers based on the CFASS. 

The evaluation found no significant differences in reading and math achievement between 
students in treatment and students in control schools, between students in treatment schools with 
medium or high fidelity and students in control schools, or between students who had an 
ECTFSI teacher and students who had a matched comparison teacher. 

While the full implementation of the FMTI model could not be tested through this evaluation, 
there are many lessons it can provide for other teacher quality improvement efforts. For example, 
the evaluation elucidates the influence of district and state mandates for turning around low 
achieving schools and how these mandates can conflict with recommended schoolwide and 
classroom practices. It shows how difficult it is for principals to find time outside of their school 
duties to engage in professional learning, but the value they find in those learning opportunities 
when they are able to engage in it. It also shows the demand teachers have for professional 
learning that they find applicable to their instructional practices and that fosters collaboration. 
Finally, it shows how high-quality facilitation, materials, and processes enable productive 
teacher learning. 

The FMTI evaluation was not sufficiently robust to definitively determine the effectiveness of 
the program. However, pockets of positive, significant findings suggest that the initiative may 
have potential to change instructional practices, although the changes in instructional practices 
may not be strong or relevant enough to change student achievement. Importantly, the evaluation 
has illuminated lessons gleaned from FMTI about how to effectively provide job-embedded 
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professional development—a model that still holds great promise—that can be used to support 
other teacher quality improvement efforts. 
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Appendix A. Methods 


The methods used for the formative, implementation fidelity, schoolwide impact, and ECTLSI 
impact evaluations are described below and displayed in a timeline in Exhibit A-l. 

Formative Evaluation 

In Years 1 and 2 of FMTI’s implementation, SRI conducted a formative evaluation to learn about 
the program’s implementation, accomplishments, challenges, and strengths to support program 
refinement, replication, and the sharing of lessons learned with the field of education. The 
formative evaluation collected data through district and school staff interviews and a review of 
program documents. 

SRI conducted semi-structured interviews in spring of 2012 and fall of 2013 with key 
informants, including principals, teachers participating in the Teacher Fellows and ECTLSI 
program, district staff managing the initiative, and UF faculty. At the school level, interview 
topics included professional background, participation in the various FMTI programs (e.g., 
Principal Fellows, summer Leadership Institute, and graduate degree program), challenges to 
participation, perceived impacts and benefits of the program on staff and students, teacher 
community and collaboration, and kindergarten transition and parent outreach. At the initiative 
level, interviews focused on supports and challenges for implementation, perceived impacts of 
the initiative on participating schools and the district as a whole, coordination and collaboration 
across initiative partners, as well as plans for sustainability. 

SRI researchers conducted interviews in six of the 20 schools participating in FMTI. Sampled 
schools represented the four geographic regions of the M-DCPS district and varied in levels of 
participation in the FMTI (e.g., from schools that had several teachers in the ECTLSI graduate 
program to schools that had no teachers in it). In spring 2012, we interviewed 19 teachers, 
including eight teachers enrolled in the ECTLSI graduate degree program, and all the school 
principals. In fall 2013, we interviewed 23 teachers, including ten teachers enrolled in the 
ECTLSI graduate degree program, and all the school principals. The ECTLSI graduate degree 
program teachers were equally representative of teachers who started the program in 2011 
(cohort 1) and those who started in 2012 (cohort 2). We also interviewed district staff managing 
the initiative and UF faculty, including professors-in-residence and developers of the ECTLSI 
graduate degree courses. 

The formative evaluation also included a review of program documents, including “The Link,” a 
periodic newsletter about FMTI published by UF, the Learning Showcase program, and ECTLSI 
course curricula. 

Fidelity of Implementation 

The implementation study examined the research question: What is the fidelity of 
implementation of the Florida Master Teacher Initiative for each key program component? To 
measure participation and fidelity, SRI worked with the program developer, the University of 
Florida, to identify types and intensity of activities in each of the program components believed 
necessary to bring about the desired changes and outcomes outlined in the program logic model. 
These assumptions were used to develop the fidelity measure included in Appendix B. SRI 
collected administrative data on participation in the various program components. The fidelity 
measure and findings are included in Appendix B. 
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Exhibit A-1. Data Collection Schedule for 5-year Project 



Year 1 

Year 2 

Year 3 

Year 4 

Year5 


2010 

2011 

2011 

2012 

2012 

2013 

2013 

2014 

2014 

2015 


F 

w 

Sp 

S 

F 

W 

Sp 

s 

F 

W 

Sp 

S 

F 

W 

Sp 

s 

F 

W 

Sp 

s 






Cl: Graduate Program Cohort 1 (Sept 2011 - May 2014) 















C2: Graduate Program Cohort 2 (Sept 2012 - May 2015) 


Interviews 







• 

Apr-May 


• 

Oct 












Teacher surveys 




• 

Aug 













• 

Aug 




Teacher observations 





• 

Cl 

Oct 




• 

C2 

Oct 








• 

C1&2 

Oct 




Participation data 








• 




• 




• 





Student achievement analysis 

















• 





*F = October, November, December; W = January, February, March; Sp = April, May, June; S = July, August, September 
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Fidelity was measured in all 20 schools implementing the intervention. To construct fidelity 
scores, data were collected on teachers’ participation in the Teacher Fellows Program; teachers’ 
participation and performance in the ECTLSI program; principals’ participation in the Principal 
Fellows program; and administrators’ and teachers’ participation in Summer Leadership 
institutes. Fidelity scores were used to assess and report whether each key component of the 
Florida Master Teacher Initiative components were implemented with fidelity in at least 75% of 
the treatment schools. 

Schoolwide Impact 

The FMTI program was conceptualized as a full-school intervention. Thus, SRI researchers 
designed the schoolwide impact study to examine the effect of FMTI on all teachers and students 
in the target grade levels (pre-kindergarten through fifth grade) of treatment schools. The 
schoolwide impact study includes both confirmatory analyses that include all schools and 
students randomized to treatment and control conditions and exploratory analyses that focus on 
estimating the effect of the program under ideal conditions such as in medium- to high fidelity 
schools or for students who remained in the same school for the full, three-year intervention 
period. In this section, we describe the initial school sample selection, random assignment 
procedures, and data collection and measures for the schoolwide impact study. Then we describe 
the analytic approach for each set of analyses. For analyses with high attrition we also present 
measures of baseline equivalence. 

Sample Selection 

Miami-Dade County public elementary schools were eligible for the study if they were Title I 
schools with a pre-kindergarten program, had at least four teachers interested in the graduate 
program, and had no previous experience with FMTI professional development opportunities 
through a similar effort, Ready Schools Miami. 

The district held a meeting with principals to explain the criteria for participation and sent letters 
to all the principals inviting them to participate with the goal of recruiting 50 schools. The total 
elementary schools and K-8 centers that were eligible to participate in FMTI was 133. A total of 
40 elementary schools, 10 short of the initial goal, agreed to participate in the study and to 
participate even if they were assigned to the control rather than the treatment condition. The 
control schools received the status-quo professional development offered by the school district 
and received $1,500 for their participation in the evaluation. 

Random Assignment of Schools 

Prior to randomization, the 40 study schools were blocked by region and also by voting district 
within region if a region-voting district block had more than one school. There are five regions 
and nine voting districts in the Miami-Dade County school district, defined by location and 
feeder patterns. The racial/ethnic composition of the student populations varied across regions; 
however, study schools in all regions were Title I schools. Among the regions and voting 
districts covered by the 40 study schools, some region/voting district combinations only had one 
school each, in which case we ignored the voting district in blocking. Using this strategy we 
came up with 13 blocks. Schools within each block were randomly assigned to either the 
intervention or control condition, with 20 schools assigned to the intervention condition and 20 
schools assigned to the control condition. 
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Because random assignment with only 40 schools cannot guarantee well-balanced treatment and 
comparison schools in all important school characteristics, we conducted the randomization of 
schools six times and selected the assignment that yielded the best balanced groups. For each of 
the six randomization outcomes, we compared school demographics in both conditions (e.g., 3 ld 
grade reading and math achievement; ethnic composition; percent eligible for free and reduced 
price meals; percentage of dual language learners; school size). Test scores in third grade reading 
and math were given the most weight in determining group equivalence. The outcome that 
yielded the most equivalent intervention and control groups was selected for the study. 

Measures and Data Collection for Schoolwide Impact Evaluation 

To assess impacts on teachers and students, we collected teacher surveys and analyzed student 
standardized test performance data in both treatment and control schools. 

Teacher surveys. SRI conducted a survey of all teachers in both intervention and control 
schools at baseline (late summer/early fall 2011) and at post-test (late summer/early fall 2014). 

Target sample: SRI administered the teacher survey to all teachers at a school with instructional 
responsibilities who had taught at the same school the prior year. Following survey 
administration, we removed non-classroom teachers (e.g. media specialists, instructional 
coaches, special area teachers) and administrators from the survey sample and any teachers who 
had not taught at the school during the prior year. Only Pre-K through 5 th grade classroom 
teachers were included in the final sample. 

Administration procedures. Prior to administration, rosters for teaching staff were collected for 
all schools. For the baseline survey, SRI prepared a survey packet for each teacher listed on the 
school roster and provided extra surveys for new teachers. SRI worked with the principal at each 
school to identify a person who would serve as the survey administrator. This person received 
the survey materials, distributed the surveys to teachers, collected completed surveys in sealed 
envelopes, tracked which teachers returned surveys, entered the names of new teachers on a 
roster tracking sheet, and mailed all surveys (both completed and noncompleted surveys) back to 
SRI. SRI provided a $50 gift certificate in appreciation for the survey administrator’s help. Most 
surveys were conducted during staff meetings that took place on the two teacher workdays prior 
to students beginning school. Some schools distributed the surveys at meetings early in the 
school year. All schools completed baseline surveys by late September 2011. The majority of 
schools returned follow-up surveys by early October 2014. One school did not complete the 
survey until early November 2014. The survey asked teachers about the prior school year; 
therefore, the later completion of the surveys should not have made a significant difference. 
Exhibit A-2 shows the response rate for 2011 and 2014 for the target teachers. 
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Exhibit A-2. Teacher Survey Response Rate 



2011 

Treatment 

2011 

Control 

2014 

Treatment 

2014 

Control 

Total survey eligible respondents 

685 

712 

643 

692 

Total number of respondents 

596 

631 

565 

634 

Responses rate 

87.0% 

88.2% 

87.9% 

91.6% 


School culture and instructional practice measures. The survey asked teachers to rate statements 
about school culture and their instructional practices on 4- and 5-point Likert scales (e.g., the 
extent to which they agree/disagree with statements and the frequency of specific practices). 
From the post-test teacher survey, measures of school culture and instructional practice were 
constructed using the constructs named in the FMTI logic model as a guide. Construction of the 
final school culture and instructional practice measures were based on a statistical analysis of the 
pre-test teacher survey data. Those factors on the pretest survey with an internal consistency 
score (i.e., Cronbach Alpha) of greater than .7 were deemed reliable measures. 


Exhibit A-3. Survey Teacher Outcome Measures 


Construct 

Cronbach’s 
Alpha, std 

Survey Items 

Frequency of 
Collaboration 
around 

Instruction 

0.90 

Mean of the following: 

Discussed what I/they learned at a workshop or conference. 
Shared, discussed, and analyzed student work. 

Discussed and analyzed instructional issues and problems. 

Shared and discussed research on effective teaching methods. 
Developed teaching materials or activities for particular classes. 
Discussed student assessment data to make decisions about 
instruction. 

Culturally 

Responsive 

Instruction 

0.89 

Mean of the following: 

Adapted lessons from texts or curriculum guides to the cultural 
background of my students. 

Provided opportunities for students to value and explore diversity 
(such as cultural heritage) in themselves and others. 

Used instructional activities that build on home 

Used Classroom materials that reflect the background and 
experiences of my students. 

Differentiated 

Instruction 

0.74 

Mean of the following: 

Used a wide variety of instructional strategies that are related to 
different learning styles. 

Provided multiple ways for students to demonstrate knowledge 
and skills. 

Used pre-assessments of students' skills to plan instruction. 
Determine students’ interests to help connect learning to their 
specific interests. 

Assessment - 
informed practice 

0.80 

Mean of the following: 

Individualize instruction for each student. 

Inform curricular and lesson planning. 

Evaluate the overall effectiveness of my instructional practice. 
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Construct 

Cronbach’s 
Alpha, std 

Survey Items 

Developmental^ 

Appropriate 

Practice 

0.89 

Mean of the following: 

Use manipulatives, real objects (e.g., plants, animals), and 
concrete materials as part of their learning experiences. 

Engage in inquiry through experiments or projects engage in 
open exploration or play. 

Listen, sing, and/or move to music as part of my lessons. 
Represent what they learn in ways other than writing (art, 
constructions, dramatizations). 

Family 

Partnerships 

0.82 

Mean of the following: 

Called or sent a personal note/email to discuss a concern. 

Called or sent a personal note/email to share positive news. 

Talked with them informally before or after class. 

Sent home activities for them to do with their children to support 
student learning. 

Invited them to help out in my classroom. 

Used school-based resources such as a Community 

Involvement Specialist to reach out to families. 

Emphasis on 
Higher-Order 
Thinking Skills 

0.89 

Mean of the following: 

Discuss possible solutions to problems with other students. 

Think aloud as they tried to solve a problem. 

Apply content knowledge to real-world scenarios. 

Discuss their point of view about something they read or 1 read to 
them. 

Discuss connections between a reading and real-life people or 
situations. 

Generate a prediction or hypothesis. 

Think about the factors that influenced an idea or caused an 
event to happen. 

Learner- 

Centered 

Instruction 

0.77 

Mean of the following: 

Make choices about their own activities (what they did and/or 
how they did it). 

Discuss their ideas and learning with other students in formal 
groupings. 

Participate in class meetings or discussions to share feelings, 
solve problems together, or talk about personal interests. 

Teacher 

Leadership 

0.87 

Mean of the following: 

Developed curriculum to be used by a team of teachers. 

Assisted in the design or planning of staff development activities. 
Led staff development activities. 

Facilitated text-based study groups. 

Participated in peer observation, coaching, or modeling activities. 
Facilitated teacher meetings (e.g., grade level, faculty meetings, 
professional learning communities). 

Principal 

Leadership 

0.92 

Mean of the following: 

Set high standards for teaching. 

Communicated a commitment to high-quality Pre-K-3 learning. 
Actively monitored the quality of teaching in this school. 

Ensured that teachers had dedicated time for collaboration. 
Encouraged school staff to take on leadership roles. 

Supported successful transitions of students from one grade to 
the next (including from Pre-K to K). 
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Construct 

Cronbach’s 
Alpha, std 

Survey Items 

Trusting 

Relationships 

Between 

Teachers 

0.93 

Mean of the following: 

Felt supported by colleagues to try out new ideas. 

Trusted each other. 

Felt responsible to help each other do their best. 

Felt comfortable sharing their challenges with each other. 

Were open to advice and feedback from their peers. 

Early Childhood 

Teaching 

Knowledge 

0.78 

Mean of the following: 

Early child development from birth to age 5 

Child assessment 

Strategies for promoting family engagement 

Florida Early Learning and Developmental Standards for 4-years 
olds. 

General 

Instructional 

Knowledge 

0.85 

Mean of the following: 

Strategies for using ongoing student assessment to plan 
instruction. 

Strategies to create a positive learning environment. 

Strategies for integrating curriculum. 

Governance 

activities 

N/A 

If a teacher indicated the participated in one of the activities below, the 
value of this measure =1. Otherwise the value of this measure = 0. 

Mean of the following: 

Worked with preschool programs or family care centers in my 
community to promote school readiness activities. 

Promoted linkages between feeder early learning programs and 
my elementary school. 

Advocated for early learning programs and my elementary 
school. 

Spoke at school board meetings to advocate for policies, 
programs, or funding that promote child well-being. 

Outreach 

activities 

N/A 

If a teacher indicated the participated in one of the activities below, the 
value of this measure =1. Otherwise the value of this measure = 0. 

Mean of the following: 

Held a designated leadership role in the school. 

Helped decide how discretionary school funds should be used. 
Helped develop the school improvement plan. 

Regular Use of 
Variety of 
Assessments 

N/A 

Count of the number of the following activities teacher did Once or 
twice a month or more 

Mean of the following: 

Direct observation looking for specific skills 

Direct assessment or testing (e.g., district tests or chapter tests). 
Ongoing formative assessment (progress monitoring). 

Portfolios of students' work samples. 
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Teacher characteristics data. From the surveys we collected teacher characteristic information 
such as gender, ethnicity, years of teaching experience, certification type, certification area, 
highest degrees earned, whether a teacher is a special education teacher, and grade level taught. 
The variables were used as covariates in the subsequent analysis. 

Student achievement data and student teacher link data. To examine the impact of the FMTI 

on reading and math achievement for students in kindergarten to grade 5, we collected student 
achievement on standardized tests from the spring prior to the start of the program (spring 2011) 
and for each spring that the program was implemented (spring 2012, 2013, 2014). The school 
district administers the Stanford Achievement Test-Tenth Edition (SAT-10) to students in 
kindergarten through grade 2 4 and administers the Florida Comprehensive Achievement Test 
(FCAT) to students in grades 3 - 5. In the final year of the study, we obtained data files 
containing the SAT-10 and FCAT scores for all students in grades K-5 ever enrolled in treatment 
and control study schools during the year prior to the start of the program and the three years of 
the study from the district. 

To link students with specific teachers for reading and math achievement, we obtained course¬ 
taking data for treatment and control schools that connected students to individual teachers 
through courses the students took. From the course-taking data we were able to distinguish 
which teachers teach English or math for what students, so we were able to later attribute student 
achievement in reading or math to corresponding teachers. 

Student and school covariate data. We collected data on student and school characteristics to 
use as covariates in the analytic models to improve the precision of impact estimates. 

Student characteristics data. In the last year of the study, we obtained from the district student 
characteristics data on all students in grades K-5 ever enrolled in the study schools during the 
year prior to the start of the program and the three years of the study. Data on student 
demographics and other baseline student characteristics included gender, ethnicity, free lunch 
status, and English learner and special education statuses. 

School characteristics data. We collected school characteristics data, including school size, 
accountability rating, and percentages of minority students, students receiving free or reduced 
price meals, and English learning students from the district. 

Confirmatory Analysis of Schoolwide Impact on Teachers 

This section describes the analysis of teacher survey outcomes from schools randomly assigned 
to treatment and control conditions to estimate the schoolwide impact of FMTI on school culture, 
teacher leadership, and classroom practice. We included all pre-kindergarten through 5th grade 
classroom teachers with completed post-treatment surveys in the analysis, regardless of whether 
they were present at baseline (i.e., we include “joiners”). We selected this approach for the 
schoolwide impact analyses because we were interested in the program impact on school culture 
and classroom practice for the school overall, regardless of normal teacher mobility. 

Analytic approach. To assess the impact of FMTI on school culture and schoolwide teacher 
leadership and classroom practice, we analyzed teachers responses to the August 2014 survey, 


4 The school district did not administer SAT-10 to kindergarten students in Spring 2011. 
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controlling for teacher and school background variables as well as the corresponding school 
average baseline outcome from the August 2011 survey. We adjust for school-average baseline 
survey measures instead of individual teacher survey measures because the post-test teacher 
survey was completed by a somewhat different group of teachers from those who completed the 
baseline teacher survey due to teacher mobility and nonresponse to the survey. 

To address the nested nature of the data, we applied a two-level hierarchical model with teacher 
and school levels. Because we randomized schools within blocks, we also include block 
indicators in the model to control for block effects. We used a dummy variable imputation 
approach 5 to handle missing covariate data, where we set missing values for covariates to 0 and 
created a dummy variable indicator for each covariate (with a value of 1 for cases with a missing 
value and a value of 0 for cases with a non-missing value). Both the recoded covariates and the 
dummy variable indicators are included in the impact model shown below. 

Teacher-level model: 

Yij= Po/ 

+ P dj (d-th covariate at teacher level) ,y 
+ e, 7 

School-level model: 

Po/= Yoo + Yoi (Treatment); + yo2 (School average baseline outcome); + yo k (k-th covariate at 

school level); + Yom (m-th block indicator) y + U; 


Where 


Yjj is the value of the outcome variable for teacher i in school j. 

Yoi indicates the impact of school being assigned to the treatment on the teacher outcome 
controlling for teacher and school-level covariates. 


5 Puma et al (2009) have found that, in studies with a random assignment design, this approach to handling missing 
data does not bias the impact estimate or its standard error and enables cases with missing covariates to be retained 
in the analysis. Although the dummy variable adjustment method leads to biased estimates of the coefficient for the 
variable itself (Jones, 1996), the impact estimate will be unbiased if assignment to condition is uncorrelated with the 
covariate in question (Puma, Olsen, Bell, & Price, 2009). 
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Exhibit A-4 presents the full list of teacher and school covariates that were included in the 
analysis. 

Exhibit A-4. Covariates for Schoolwide Teacher Survey Impact Analysis 


Covariate 

Teacher level 

Black 

Hispanic 

Asian 

Other ethnicity 

Years of teaching experience 

Florida Temporary Certification 

National Board Certification 

Certification area: Pre-kindergarten/primary education 

Certification area: Elementary education (K-6) 

Master's degree or higher 

Grade level indicators 

Special education teacher 

School level 

Percent students with limited English proficiency 

Percent students with free or reduced price lunch status 

Percent minority students (Black and Hispanic) 

Average baseline survey measure 

Average baseline third-grade reading score 

Average baseline third grade math score 

State accountability grade 

Block indicators 


Exploratory Analysis of Schoolwide Impact on Teachers in Medium- to High- 
fidelity Schools 

This section describes the exploratory analysis on the impact of FMTI in treatment schools that 
attained medium to high fidelity to the FMTI model. 

Analytic approach. To explore the possible impact of FMTI on teachers in treatment schools 
that met the implementation threshold, we replicated the schoolwide impact analysis described 
above on a sample that included only teachers in medium- or high-fidelity treatment schools (11 
schools) and teachers in comparison schools within the same randomization blocks (18 schools). 
Randomization blocks that contained no medium- or high-fidelity schools were excluded from 
the analysis. The analytic model and set of covariates was the same as for the confirmatory 
schoolwide teacher survey analysis. 
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Confirmatory Analysis of Schoolwide Impact on Students 

This section describes the analyses of the schoolwide impact of FMTI on students who were in 
treatment and control schools prior to randomization. This is an intent-to-treat analysis, which 
means that students are retained in the analytic sample as long as they have post-treatment test 
scores and that students are classified according to their initial treatment assignment, regardless 
of whether they change schools later during the implementation of the intervention. Because all 
treatment and control schools remained in their randomly assigned status during the intervention 
period, there was no school level attrition. Furthermore, as Exhibit A-5 shows, among students in 
these schools at baseline, only 20% in treatment and 20% in control schools did not have 
outcome scores for both reading and math analyses. By What Works Clearinghouse (WWC) 
standards, this analysis is considered to be an RCT with low attrition and does not require 
evidence of baseline equivalence to meet WWC evidence standards without reservations. This 
allowed researchers to pool students across grade levels that do and do not have baseline test 
score data to estimate an overall estimate of the impact of FMTI on student achievement. 

Exhibit A-5 Sample Size and Attrition for School-wide Student Impact Analysis 



Baseline 

Student 

N 

Reading 

Student 

N 

Reading 

Attrition 

% 

Math 

Student 

N 

Math 

Attrition 

% 

Treatment 

6672 

5320 

20 

5313 

20 

Control 

6575 

5290 

20 

5274 

20 


Analytic approach. To examine the three-year impact of FMTI on all students who were in Pre- 
K through grade 2 at baseline, we pooled students at all grade levels in the same analysis. The 
outcome measures for the student achievement analyses are test scores in math and reading in 
2013-14, three years after the intervention started. Because both SAT-10 and FCAT have 
developmentally scaled scores that are comparable across grades, we standardized the 2013-14 
outcome test scores by student baseline grade level in 2010-11. Therefore even if a student was 
retained in grade, he was still compared with his original cohort at baseline. For reading and 
math separately, we calculated a z-score by taking the difference between each student’s original 
test score and the mean score for his cohort in the whole district, then dividing by the student- 
level standard deviation of the score for his cohort. 

Baseline measures of the outcomes come from student reading and math test scores in the spring 
of 2010-11, when students in first grade took the SAT-10 and students in second grade took the 
FCAT. We calculated z-scores at these grade levels for reading and math separately, using the 
same approach as for the outcome scores. Students in grades Pre-K and K at baseline did not take 
any baseline assessment, therefore do not have baseline test scores. 

We used a two-level HLM model with student and school levels for the analysis, adjusting for 
baseline student achievement, demographic characteristics, and school characteristics at their 
respective levels. Because we randomized schools within blocks, we also included block 
indicators in the model to control for block effects. We used the same dummy imputation 
approach to handling missing covariate data that is described in the schoolwide teacher impact 
analysis section. This includes imputation of baseline achievement data for all students who were 
in grades Pre-K and K at baseline. 
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Student-level model: 

Yjj = P o/ + P ij (Baseline test score) y 

+ P dj (d-th covariate at student level) y 
+ e,y 

School-level model: 

Po ,j= Yoo + Yoi (Treatment); + Yo/c (k-th covariate at school level); + Yom (m-th block indicator); 

+ U; 


Where 


Yy is the value of the 2013-14 reading/math test score for student i in school j. 

Yoi indicates the impact of schools being assigned to the treatment on the student outcome 
for students after three years, controlling for student and school-level baseline covariates. 

Exhibit A-6 presents the full list of student and school covariates that were included in the 
analysis. 


Exhibit A-6. Covariates for School-wide Impact Analysis 


Covariate 

Student level 

Female 

Black 

Hispanic 

Asian 

Other 

Free-reduced price lunch status 

Limited English proficiency 

Special education 

Grade level indicators 

Baseline test score 

School level 

Percent students with limited English proficiency 

Percent students with free or reduced price lunch status 

Percent minority students (Black and Hispanic) 

Average baseline third grade reading score 

Average baseline third grade math score 

State accountability grade 

Block indicators 
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Exploratory Analysis of Schoolwide Impact on Students who Remained in 
the Same School 

This analysis examined the impact of FMTI on students who remained in the same school during 
the three years of intervention (impact of treatment on the treated). As displayed in Exhibit A-7, 
among students in grades Pre-K through 2 at baseline, 47% in treatment and 43% in control 
schools were not retained in the sample either due to missing data or because they did not remain 
in the same school for the full three-year intervention period. We conducted the analysis for 
students in grades Pre-K to K at baseline separately from those who were in grades 1 and 2. The 
analyses are conducted separately because of differences between the two samples in the 
availability of measures for establishing baseline equivalence. For the sample of students in 
grades Pre-K to K at baseline, there are not pretest measures for the same students as those in the 
analytic sample; therefore we use baseline scores for the earlier adjacent cohort of students (from 
spring 2011 and measured in the same grade levels as the posttest - grades 2 and 3) to establish 
baseline equivalence. For the sample of students who were in grades 1 and 2 at baseline, there 
are pretest measures for the same students as those in the analytic sample, which we use to 
establish baseline equivalence. 


Exhibit A-7. Sample Size and Attrition for School-wide Impact Analysis of 
Students who Remained in the Same School 



Baseline 

Student 

N 

Reading 

Student 

N 

Reading 

Attrition 

% 

Math 

Student 

N 

Math 

Attrition 

% 

Treatment 

6672 

3,567 

47 

3,564 

47 

Control 

6575 

3,750 

43 

3,739 

43 


Analytic Approach. We conducted two separate analyses, both of which included the same 
covariates and applied the same two-level HFM model as in the above confirmatory schoolwide 
analysis. However, for the analysis of students in grades PreK to K at baseline, instead of 
imputing baseline test scores, we used school-average baseline test scores for grades 2 and 3 in 
spring 2011 as the pre-test measure. For the analysis of students in grades 1 and 2 at baseline, 
individual student pretest scores from spring 2011 were included in the model for all students in 
the analytic sample. Moreover, we excluded students with missing values on any other covariates 
for both analyses. 

Baseline equivalence. For the analysis of students who were in grades 1 and 2 at baseline 
(grades 4 and 5 at post-test), we checked baseline equivalence of reading and math scores 
respectively, using t-tests on individual student scores to assess whether there were any 
statistically significant differences between treatment and control students. For the analysis of 
students who were in Pre-K and K at baseline (grades 2 and 3 at post-test), we checked the 
equivalence of reading and math scores for the earlier, adjacent cohort of students in grades 2 
and 3 at baseline (spring 2011) using t-tests on individual student scores. 

Although the estimated treatment-control differences were statistically significant, none 
exceeded .25 standard deviations (Exhibit A-8). We therefore considered students in treatment 
and control groups to be not substantially different at baseline. In the impact model, we adjusted 
for the relevant baseline test score and other covariates as discussed above, but did not perform 
any additional weighting. 
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Exhibit A-8. Baseline Equivalence for Schoolwide Impact Analysis of Students 

who Remained in the Same School 


Student outcome measure 

Standardized 
Mean Difference 

SE 

P-value 

Student 

N 

School 

N 

Math achievement, grades Pre-K & K 

-0.086 

0.031 

0.005 

3901 

40 

Reading achievement, grades Pre-K & K 

-0.153 

0.031 

0.000 

3903 

40 

Math achievement, grades 1-2 

-0.124 

0.027 

0.000 

4732 

40 

Reading achievement, grades 1-2 

-0.118 

0.028 

0.000 

4740 

40 


Exploratory Analysis of Schoolwide Impact on Students Who Remained in 
Medium- to High-fidelity FMTI Schools 

Among the 20 schools randomly assigned to the treatment condition, 11 treatment schools are 
considered to have medium or high implementation fidelity. This exploratory analysis compared 
students who stayed for the full three-year intervention period in these 11 FMTI schools with 
students who stayed for the same time period in the 18 control schools in the same randomization 
blocks. As displayed in Exhibit A-9, among students in grades Pre-K through 2 at baseline, 65% 
of students in treatment schools and 45% in control schools did not remain in the analytic 
sample. We adopted the same analytic approach as the exploratory schoolwide impact analysis 
of students who stayed at the same school (described above) and conducted analysis separately 
for students in grades Pre-K to K at baseline (using baseline measures for the prior adjacent 
cohort of students in study schools in 2011) and those who were in grades 1 and 2 (using pretest 
measures for the same students as those in the analytic sample). 


Exhibit A-9. Sample Size and Attrition for School-wide Impact Analysis of 
Students Staying in Medium- to High-fidelity Treatment Schools and Comparison 

Schools 



Baseline 

Student 

N 

Reading 

Student 

N 

Reading 

Attrition 

% 

Math 

Student 

N 

Math 

Attrition 

% 

Treatment 

6672 

2,309 

65 

2,307 

65 

Control 

6575 

3,625 

45 

3,614 

45 


Analytic approach. We used the same covariates and applied the same two-level HLM model 
as in the above exploratory analysis with students who stayed in the same school during the three 
years of intervention. 


Baseline equivalence. For the analysis of students who were in grades 1 and 2 at baseline 
(grades 4 and 5 at post-test), we checked baseline equivalence of reading and math scores 
respectively, using t-tests on individual student scores to assess whether there were any 
statistically significant differences between treatment and control students. For the analysis of 
students who were in Pre-K and K at baseline (grades 2 and 3 at post-test), we checked the 
equivalence of reading and math scores for the earlier, adjacent cohort of students in grades 2 
and 3 at baseline, in spring 2011, using t-tests on individual student scores (Exhibit A-10). 

Although three out of four of the estimated treatment-control differences were statistically 
significant, none exceeded .25 standard deviations. We therefore considered students in 
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treatment and control groups to be not substantially different at baseline. In the impact analysis 
model, we adjusted for the relevant baseline test score and other covariates, but did not perform 
any additional weighting. 


Exhibit A-10. Baseline Equivalence for Students in Medium- to High-Fidelity 
Treatment Schools and Comparison who Remained in the Same School 


Student outcome measure 

Standardized 
Mean Difference 

SE 

P-value 

Student 

N 

School 

N 

Math achievement, grades PK-K 

-0.001 

0.036 

0.988 

3050 

29 

Reading achievement, grades PK-K 

-0.100 

0.036 

0.005 

3052 

29 

Math achievement, grades 1-2 

-0.061 

0.031 

0.050 

3859 

29 

Reading achievement, grades 1-2 

-0.067 

0.032 

0.033 

3867 

29 


ECTLSI Graduate Program Teacher Impact 

Although the FMTI program was conceptualized as a schoolwide intervention, teachers who 
participated in the ECTLSI graduate program experienced a much more intensive treatment than 
those who did not. To reflect this, SRI researchers incorporated an embedded, quasi- 
experimental design (QED) into the overall evaluation to examine the impact of ECTLSI on the 
teachers who participated in it. We estimated the effect of the ECTLSI program on teachers 
directly through the teacher survey and classroom observations, and we also estimated the added 
benefit to students of being taught by an ECTLSI teacher on top of the schoolwide FMTI effect. 
In this section, we describe sample selection for ECTLSI teachers, their matched comparison 
groups, and the additional data collection and measures collected beyond those collected through 
the schoolwide impact study. Then we describe the analytic approaches used for each data source 
(i.e., teacher survey, CLASS observations, and student achievement data) and measures of 
baseline equivalence for each set of analyses. 

Sample Selection 

Once a school was assigned to the intervention condition, all teachers of grades Pre-K to 3 were 
allowed to apply to the ECTLSI program; however, they had to be admitted to the graduate 
program based on their undergraduate GPA, GRE scores, and their application narrative. 
Professors-in-residence made presentations at many of the treatment schools and held 
information sessions at restaurants to encourage teachers in the treatment schools to apply. 
Teachers interested in applying were offered, at no cost, GRE preparation materials and training. 

A total of 50 teachers in the treatment schools submitted complete applications to Cohort 1 of the 
master’s program and 37 of those teachers were admitted (74%), 2 were allowed to take a first 
course while they retook their GREs (4%), and 11 teachers were officially denied (22%). SRI 
only evaluated those teachers who were fully admitted to the program. After receiving 
notification of acceptance into the program but prior to the start of the program, two teachers 
decided not to participate. These two teachers are not considered ECTLSI students for purposes 
of the evaluation. The program further accepted 24 Cohort 2 teachers in the second year of the 
intervention. Because the QED was based on having 50 ECTLSI teachers and there were only 35 
in Cohort 1 by the start of the second year of implementation, we randomly selected 15 
additional teachers from Cohort 2 in 2012-13 for inclusion in the classroom observations. We 
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included all ECTLSI teachers with complete data in the survey and student achievement 
analyses. Because the analytic strategy for identifying a comparison group for ECTLSI teachers 
depended on the data source and its specific analytical approach, we discuss this aspect of the 
sample selection within the analytic approach section for each data source below. 

Measures and Data Collection for Analysis of ECTLSI Impact 

For the embedded QED, in addition to collecting the same data that were collected on teachers 
and students for the schoolwide RCT, the evaluation conducted classroom observations in the 
classrooms of teachers in the ECTLSI graduate program and of matched comparison teachers in 
controls schools. To identify a matched comparison group for observations, we used propensity 
score matching. We first included all Pre-K-3 teachers in treatment schools to posit a model 
predicting the probability of being in the master’s program based on teachers’ indication of 
interest to participate in the program in spring 2011, being a special education teacher, years of 
teaching experience, academic degrees earned, whether the teacher has national board 
certification, areas of certification, ethnicity, and scores on six classroom practice scales from the 
baseline teacher survey (learner-centered instruction, assessment-informed practice, 
developmentally-appropriate instruction, higher-order thinking skills, differentiated instruction, 
and culturally-responsive instruction). We then extrapolated the estimated propensity score 
model to teachers in comparison schools and calculated propensity scores for them as well. 

We applied one-to-one nearest neighbor matching without replacement to select a comparison 
teacher in control schools for each of the 35 Cohort 1 ECTLSI teachers and a randomly selected 
pull of 15 Cohort 2 ECTLSI teachers. These matched comparison teachers were the highest 
priority matches. We then matched each of the 50 ECTLSI teachers with three additional 
comparison teachers (one-to-three nearest neighbor matching, with replacement) as backup 
matches in case the main comparison teachers refused to participate in the study. We 
subsequently produced prioritized lists of matches for each ECTLSI teacher and proceeded to 
contact comparison teachers in order of the quality of the match until we found one who was 
willing to participate. 

Classroom observations. To assess changes in teaching skills, SRI conducted repeated 2-3 hour 
classroom observations (i.e., a minimum of four 20 minute cycles of observation as 
recommended by the assessment’s developer) using the Classroom Assessment Scoring System 
(CLASS) observation tool with the Pre-K-3 teachers participating in the first and second cohort 
of the graduate program and matched teachers in the control schools. The CLASS was used to 
measure outcomes in each of the following domains: emotional support, classroom organization, 
and instructional support. Emotional support includes measures of positive climate, negative 
climate, teacher sensitivity, and regard for student perspectives. Classroom organization includes 
measures of behavior management, productivity, and instructional learning formats. Instructional 
support includes measures of concept development and quality feedback. 

Observations were conducted in the fall of students’ first year in the graduate program and two 
or three years later (depending on cohort) by trained observers who had received certification in 
the reliable use of the tool from the CLASS developer, Teachstone. Note that teachers in cohort 2 
had a year of exposure to the other components of the intervention in the school prior to starting 
the graduate program, while teachers in cohort 1 began experiencing all components of the FMTI 
at the same time. In addition, cohort 1 teachers received their post observation approximately 6 
months after completing their graduate program while cohort 2 teachers received their post 
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observation approximately 6 months before completing their graduate program. Comparison 
teachers received $100 for each observation as a token of appreciation for their participation. 
Exhibit A-12 provides the list of reasons for missing observations at follow-up. 

Measures of classroom instruction were constructed from CLASS ratings completed during 
classroom observations in classrooms of teachers in the graduate program and matched 
comparison teachers. For the CLASS observation, dimensions of classroom organization, 
emotional support, and instructional support are each rated on a 7-point Likert scale for multiple 
30-minute observation cycles. Scores for each dimension were averaged across all observation 
cycles in a classroom and then the average dimension scores were summed and averaged for 
each of the three instructional domains - classroom organization, emotional support, and 
instructional support. Exhibit A-13 provides psychometric properties for the CLASS observation. 
Impacts of FMTI on each of the three instructional outcome scores were examined for teachers 
in the graduate program and matched comparison teachers. 


Exhibit A-11. CLASS Observation Teacher Sample 



Treatment 

N 

Control 

N 

Observed at baseline 

49 

48 

Observed at follow-up/Analytic sample 

37 

25 


Exhibit A-12. Reasons for Excluding from Follow-up Analysis 



T reatment 

N 

Control 

N 

On leave during observation period 

5 

5 

No longer teaching in a Pre-K to 3 rd grade position 

2 

6 

No longer teaching within district 

1 

2 

Declined to participate 

1 

4 

Non-responsive to observation requests 

1 

5 

Unable to locate 

0 

1 

Did not remain enrolled in EC-TLSI program 

2 

0 

Total 

12 

23 
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Exhibit A-13. Domains and Psychometric Properties for CLASS Observation 


Domain 

Instrument 

reference 

Normed or 
State 
Test? 

Test- 

Retest 

Reliability 

Internal 

Consistency 

Inter-rater 

Reliability 

Score 

Classroom 

instruction: 

emotional 

support 

Pianta, 
LaParo, 
Hamre, 2008 

No 

.73-.85 

.77-.89 

.79-.97 

1-7 Likert 
scale score 

Classroom 

instruction: 

classroom 

organization 

Pianta, 
LaParo, 
Hamre, 2008 

No 

.73-.85 

.77-.89 

.79-.97 

1-7 Likert 
scale score 

Classroom 

instruction: 

instructional 

support 

Pianta, 
LaParo, 
Hamre, 2008 

No 

.73-.85 

.77-.89 

.79-.97 

1-7 Likert 
scale score 


Confirmatory Analysis of ECTLSI Impact on Teacher Survey Outcomes 

This section describes the analysis of teacher survey outcomes to estimate the impact of the 
ECTLSI graduate program on school culture, teacher leadership, and classroom practice. 

Analytic approach. To assess the impact of the ECTLSI graduate program on participants’ 
reports of school culture, teacher leadership, and classroom practice, we compared ECTLSI 
teachers’ survey responses to all Pre-K to 3 teachers in control schools who responded to both 
the baseline and post-treatment surveys. We explored the possibility of selecting a matched 
comparison group of teachers for this analysis, but found that it was not possible to select one 
sample that was balanced on all sixteen baseline survey measures. 

We analyzed teacher responses to the August 2014 survey, controlling for teacher and school 
background variables, including teachers’ individual baseline survey measures from the August 
2011 survey. We excluded teachers with missing covariate data from these analyses. To address 
the nested nature of the data, we applied a two-level hierarchical model with teacher and school 
levels. Because not all of the 20 treatment schools had ECTLSI teachers, we combined 
geographically neighboring blocks as necessary to ensure that there was always at least one 
treatment and one control school in each block. 

The analytical models used for survey outcomes were: 

Teacher-level model: 

Yij= P 0/ 

+ p dj (d-th covariate at teacher level) Sl j 

+ Cij 

School-level model: 

Po ,j= Yoo + Yoi (Treatment) y + Yo k (k-th covariate at school level) y + Yom (m-th block indicator) y 

+ Uy 
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Where 


Yij is the value of the teacher practice outcome variable for teacher i in school j. 

Yoi indicates the effect of school being assigned to the treatment on the teacher practice 
outcome. 

Exhibit A-14 presents the full list of teacher and school covariates that were included in the 
analysis. 


Exhibit A-14. Covariates for ECTLSI Teacher Impact Analysis 


Covariate 

Teacher level 

Black 

Hispanic 

Asian 

Other ethnicity 

Years of teaching experience 

Florida Temporary Certification 

National Board Certification 

Certification area: Pre-Kindergarten/primary education 

Certification area: Elementary education (K-6) 

Master's degree or higher 

Grade band indicators 

Baseline survey measure 

School level 

Percent students with limited English proficiency 

Percent students with free-reduced price lunch status 

Percent minority students (Black and Hispanic) 

Average baseline third-grade reading score 

Average baseline third grade math score 

State accountability grade 

Block indicators 


Baseline equivalence. We checked baseline equivalence of all survey measures, using t-tests on 
individual teacher measures to assess whether there were any statistically significant differences 
between ECTLSI and control teachers. We found that the estimated treatment-control difference 
exceeded .25 standard deviations for six survey measures (Exhibit A-14). For these six measures, 
we calculated propensity score weights using the inverse propensity score calculated from a 
logistic regression that included the baseline survey measure as well as teacher demographic 
characteristics. We then utilized these weights in the analytic models along with the covariates 
discussed above. For survey measures where the estimated treatment-control difference did not 
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exceed .25 standard deviations, we considered treatment and control groups to be not 
substantially different at baseline. We adjusted for the relevant baseline test score and other 
covariates, but did not perform any additional weighting. Exhibit A-15 displays both unweighted 
and weighted standardized treatment-control differences at baseline. 

Confirmatory Analysis of ECTLSI Impact on CLASS Outcomes 

This section describes the analysis of CLASS observations to estimate the impact of the ECTLSI 
graduate program on classroom instruction. For more information on the identification of the 
matched comparison group, sample attrition, and data collection procedures, see the above 
section on Measures and Data Collection for Analysis of ECTLSI Teacher Impact. 

Analytic approach. To assess the impact of the ECTLSI graduate program on classroom 
instruction we compared classroom instruction indicators obtained from CLASS observations 
between ECTLSI teachers and their matched comparison teachers. We use the same model and 
set of covariates as in the analysis of ECTLSI teacher survey outcomes described above except 
that we used teachers’ baseline CLASS observation scores as the pre-treatment measure of 
classroom instruction. Also, because not all 40 study schools had ECTLSI or matched 
comparison teachers, we combined geographically neighboring blocks as necessary to ensure 
that there was always at least one treatment and one control school in each block. 

Baseline equivalence. We checked baseline equivalence of each CLASS domain, using t-tests 
on individual teacher measures to assess whether there were any statistically significant 
differences between ECTLSI and matched control teachers. We found that the estimated 
treatment-control difference exceeded .25 standard deviations for the instructional support 
domain only (Exhibit A-16). For instructional support, we calculated a propensity score weight 
using the inverse propensity score calculated from a logistic regression that included the baseline 
instructional support measure as well as teacher demographic characteristics. We then utilized 
this weight in our analytic model along with the covariates discussed above. For the domains 
where the estimated treatment-control difference did not exceed .25 standard deviations, we 
considered treatment and control groups to be not substantially different at baseline. We adjusted 
for the relevant baseline test score and other covariates, but did not perform any additional 
weighting. Exhibit A-14 displays both unweighted and weighted standardized treatment-control 
differences at baseline. 
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Exhibit A-15. Baseline Test of Difference Between ECTLSI and Control Teachers on Teacher Survey 


Survey outcome measure 

Unweighted 

Standardized 

Mean 

Difference 

Unweighted 

SE 

Unweighted 

P-value 

Weighted 

Standardized 

Mean 

Difference 

Weighted 

SE 

Weighted 

P-value 

Teacher 

N 

School 

N 

Learner-centered instruction 

-0.391 

0.180 

0.031 

0.013 

0.205 

.949 

439 

33 

Assessment-informed practice 

0.048 

0.126 

0.704 

— 

— 

— 

440 

33 

Developmentally appropriate practices 

0.192 

0.182 

0.290 

— 

— 

— 

440 

33 

Emphasis on higher order thinking 
skills 

-0.292 

0.186 

0.118 

0.018 

0.230 

0.938 

445 

33 

Differentiated instruction 

-0.244 

0.182 

0.181 

— 

— 

— 

444 

33 

Culturally responsive instruction 

-0.274 

0.182 

0.134 

-0.014 

0.204 

0.943 

436 

33 

Collaboration around instruction 

-0.266 

0.185 

0.151 

-0.009 

0.209 

0.966 

443 

33 

Trusting relationships between 
teachers 

-0.171 

0.174 

0.324 

_ 

_ 

_ 

441 

33 

Principal leadership 

-0.667 

0.182 

0.000 

-0.112 

0.251 

0.654 

442 

33 

Teacher leadership 

-0.219 

0.169 

0.198 

— 

— 

— 

445 

33 

Family partnerships 

-0.268 

0.186 

0.150 

-0.079 

0.199 

0.690 

442 

33 

Regular use of variety of assessments 

0.143 

0.186 

0.444 

— 

— 

— 

440 

33 

Early childhood instructional 
knowledge 

-0.071 

0.175 

0.682 

_ 

_ 

_ 

443 

33 

General instructional knowledge 

-0.139 

0.174 

0.424 

— 

— 

— 

443 

33 

Governance activities* 

-0.044 

— 

— 

— 

— 

— 

446 

33 

Early learning outreach activities* 

-0.135 

— 

— 

— 

— 

— 

446 

33 


*For assessing baseline equivalence of binary outcomes, we follow a procedure adapted from the version 3.0 WWC Standards and Procedures 
manual, pp. F.5 - F.6. 
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Exhibit A-16. Baseline Test of Difference Between ECTLSI and 
Control Teachers on CLASS Observations 


CLASS 

domain 

Unweighted 

Standardized 

Mean 

Difference 

Unweighted 

SE 

Unweighted 

P-value 

Weighted 

Standardized 

Mean 

Difference 

Weighted 

SE 

Weighted 

P-value 

Teacher 

N 

School 

N 

Instructional 

support 

-0.398 

0.256 

0.125 

-0.167 

0.272 

0.542 

62 

26 

Emotional 

support 

-0.036 

0.261 

0.891 

_ 

_ 

_ 

62 

26 

Classroom 

organization 

-0.104 

0.261 

0.69 

_ 

_ 

_ 

62 

26 
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Confirmatory Analysis of ECTLSI Impact on Student Outcomes 

In this section we describe the confirmatory analysis of the added impact of participation in the 
ECTLSI program on top of the schoolwide FMTI effect for grade 1 to 3 students’ reading and 
math achievement. This analysis used a difference-in-difference approach to estimate the impact 
of the FMTI program on students’ reading and math achievement (measured by the SAT-10 and 
FCAT) for students of teachers in the master’s program (compared to matched comparison 
teachers in the control schools) over and above the impact for students in intervention schools 
whose teachers did not participate in the master’s program (compared to a comparison group of 
teachers in control schools). 

Outcome student achievement measures are student achievement test scores in math and reading 
in 2013-14, after they received one year of instruction from the four groups of teachers. As 
described previously, students in grades 1 and 2 took the SAT-10 and students in grade 3 took 
FCAT. We calculated grade-specific z-scores for reading and math separately. 

Baseline student achievement outcome measures are student achievement test scores in reading 
and math in the spring of 2012-13, when students in K - second grade took the SAT-10 6 . We 
calculated baseline grade-specific z-scores for reading and math respectively. 

Analytic approach. Among 45 eligible ECTLSI teachers for whom we had baseline survey data 
(which were used to identify matched comparison teachers in control schools), 23 teachers from 
10 schools taught English and had students with a reading test score in 2013-14, and 20 teachers 
from 8 schools taught math and had students with a math test score in 2013-14. There were 14 
teachers who taught both reading and math. 

For each ECTLSI teacher, we used 1-to-l nearest neighbor propensity score matching to find a 
matched teacher from control schools. There were three types of teachers: those who taught both 
reading and math, those who only taught English, and those who only taught math. For the 14 
teachers who taught both reading and math, we matched them with teachers in control schools 
who taught both subjects. We matched 9 ECTLSI teachers who taught only English with 
teachers in control schools who taught only English, and 6 ECTLSI teachers who taught only 
math to those in control schools who only taught math. 

For each of the three kinds of teachers, we first included ECTLSI teachers and teachers in 
control schools to posit a model predicting the probability of being in the ECTLSI’s program 
based on teachers’ indication of interest to participate in the program in spring 2011, years of 
teaching experience, academic degrees earned, certification type and area, as well as six 
classroom practice scales from the baseline teacher survey (learner-centered instruction, 
assessment-informed practice, developmentally-appropriate instruction, higher-order thinking 
skills, differentiated instruction, and culturally-responsive instruction). We then estimated the 
propensity score of being an ECTLSI teacher for all teachers, and matched each of the ECTLSI 
teachers with a teacher in a control school who taught the same grade level and had the closest 
propensity score. 


6 Florida began administering the SAT10 to Kindergarten students in 2011-12—too late for our RCT design, 
but in time for the embedded QED. 
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A total of 29 ECTLSI teachers taught either reading or math in grades 1 to 3, and a total of 260 
control teachers taught either reading or math at these grade levels. Table A-17 shows the 
variable balance for ECTLSI and control teachers before and after matching. Before matching, 
ECTLSI teachers were more likely than control teachers to express an interest to participate in 
the ECTLSI program at baseline (48% versus 15%). After matching, 41% of the matched control 
teachers expressed interest, reducing the gap from 33% to only 7%. Matching also reduced the 
gaps in classroom practice between the two groups of teachers, rendering all post-matching 
differences within .2 standard deviations on all variables. This ensures the comparison of similar 
teachers between treatment and control schools in terms of baseline teaching practices and 
motivation to participate in the program. On the other hand, matching did not close the gap in 
having a master’s degree or higher degree or eliminate differences in racial composition between 
the two groups of teachers. We further adjusted all these variables in the subsequent analytic 
model. 

Table A-17. Descriptive Information on ECTISI and Comparison School Teachers, 

Before and After Matching 



ECTLSI 

teachers 

Original 

(n=29) 

Comparison 

teachers 

Original 

(n=260) 

Comparison 

teachers 

Matched 

(n=29) 

Interest to participate in ECTLSI 

0.48 

0.15 

0.41 

Years of teaching in total 

10.59 

14.13 

13.72 

Florida temporary certification 

0.00 

0.01 

0.07 

Florida professional certification 

1.00 

0.96 

0.90 

National board certification 

0.10 

0.05 

0.10 

Elementary certification 

0.76 

0.79 

0.83 

Pre-K/ primary certification 

0.21 

0.13 

0.14 

Preschool certification 

0.00 

0.02 

0.03 

Black 

0.07 

0.24 

0.34 

White 

0.38 

0.15 

0.14 

Latino 

0.48 

0.53 

0.41 

Other Ethnicity 

0.07 

0.08 

0.10 

Having a master's degree or higher 

0.31 

0.48 

0.55 

Student Centered Instruction 

3.80 

3.96 

3.84 

Assessment-informed practice 

4.07 

4.25 

4.05 

Developmentally Appropriate Instruction 

3.98 

3.91 

3.91 

Higher Order Thinking Skills 

4.34 

4.49 

4.40 

Differentiated Instruction 

4.27 

4.46 

4.34 

Culturally Responsive Instruction 

3.51 

3.85 

3.64 
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Exhibit A-18 compares non-ECTLSI teachers in treatment schools and nonmatched teachers in 
control schools, who are similar in teacher background and classroom practice measures. 

Exhibit A-18. Descriptive Information on Non-ECTISI and Nonmatched Teachers 

in Treatment and Control Schools 



Non-ECTISI 
Teachers in 
Treatment 
Schools 
(n=216) 

Nonmatched 
Teachers in 
Control 
Schools 
(n=231) 

Interest to participate in ECTLSI 

0.17 

0.12 

Years of teaching in total 

14.71 

14.18 

Florida temporary certification 

0.01 

0.00 

Florida professional certification 

0.97 

0.97 

National board certification 

0.05 

0.04 

Elementary certification 

0.84 

0.79 

Pre-K/ primary certification 

0.09 

0.13 

Preschool certification 

0.01 

0.02 

Black 

0.29 

0.23 

White 

0.15 

0.15 

Latino 

0.44 

0.54 

Other Ethnicity 

0.12 

0.08 

Having a master's degree or higher 

0.51 

0.47 

Student Centered Instruction 

4.03 

3.97 

Assessment-informed practice 

4.38 

4.28 

Developmentally Appropriate Instruction 

3.91 

3.91 

Higher Order Thinking Skills 

4.51 

4.50 

Differentiated Instruction 

4.49 

4.48 

Culturally Responsive Instruction 

3.90 

3.88 


For the subsequent difference-in-differences analysis, we only included schools with either at 
least one ECTLSI teacher or at least one matched comparison teacher. 

Because some teachers co-teach math or English, we generated “virtual” teachers (the 
combination of all teachers teaching a class) and nested students under those virtual teachers. For 
example, if a class was taught by two different English teachers, we averaged the characteristics 
of these two teachers to generate a virtual teacher for that class. If one of the two teachers was an 
ECTLSI teacher, the virtual teacher has a value of 0.5 for the ECTLSI teacher indicator. For 
example, among 23 ECTLSI English teachers, 18 each independently taught one class only, three 
each co-taught a class with a non-ECTLSI teacher only, and two each co-taught a class with a 
non-ECTLSI teacher in addition to a class they taught independently. Therefore there are 25 
ECTLSI virtual teachers included in the analysis, 20 with a value of lfor the ECTLSI indicator 
and 5 with a value of 0.5 for the indicator. Exhibit A-19 shows the number of teachers and the 
corresponding number of virtual teachers in each group for the difference-in-differences analysis. 
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We hereafter refer to a teacher/teachers teaching a class as a virtual teacher, while the vast 
majority of virtual teachers were in fact only one teacher. 


Exhibit A-19 Number of Teachers and Virtual Teachers in Different Comparison 

Groups 



Teachers 

N 

Virtual Teachers 

N 

Reading 



Nonmatch ed 
teachers in 
control schools 

102 

97 

Matched 
teachers in 
control schools 

23 

27 

Non-ECTLSI 
teachers in 
treatment 
schools 

83 

83 

ECTLSI 

teachers 

23 

25 

Math 



Non match ed 
teachers in 
control schools 

103 

103 

Matched 
teachers in 
control schools 

20 

20 

Non-ECTLSI 
teachers in 
treatment 
schools 

59 

60 

ECTLSI 

teachers 

20 

20 


We used a three-level HLM model with student, virtual teacher, and school levels, adjusting for 
student prior test score and demographics, teacher characteristics, school characteristics and 
block indicators. We included in the analysis students who are linked to the four types of virtual 
teachers and have no missing values on the outcome score or any of the covariates. 

In the model we included an ECTLSI/matched teacher indicator, a treatment school indicator, 
and the interaction term between the ECTLS I/matchcd teacher indicator and the treatment school 
indicator. The interaction term represents the difference-in-differences estimate, providing the 
impact of the specific ECTLSI program on teachers, net of the schoolwide program impact. The 
model is shown below. 
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Student-level model: 

Y S ij = n oij + 7i P ij (Previous year test score) sij 

+ n dij (d-th covariate at student level)) ,// 

+ Zsij 

Teacher level model: 

Ttoij = Poo, + Poij (ECTLSI teacher or matched comparison) + p 0t j (t-th covariate at 
teacher level) + roij 

School-level model: 

Pooy = Yooo + Yooi (Treatment) , + YooAr (k-th covariate at school level) y + yoom (m-th block 
indicator) y + Uo; 

Po 1 / = Yoio + Yon (Treatment) , 

Where 

yooo is the average student outcome with teachers in control schools who are not matched 
with ECTLSI teachers (with all covariates grand- mean centered) 

yooi indicates the impact of school being assigned to the treatment on the student outcome 
for students with teachers not in the ECTLSI’s program. 

yoio is the average difference in student outcomes among students with control teachers in 
the matched comparison for ECTLSI teachers and among students with control teachers in the 
comparison group for teachers not in the ECTLSI’s program. 

yooi + yon is the total impact of the program on students with ECTLSI teachers. 

you is the additional impact of the program on students with ECTLSI teachers, over and 
above the treatment impact on students with teachers not in the ECTLSI’s program. 
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Exhibit A-20 presents the full list of student and school covariates that were included in the 
analysis. 


Exhibit A-20. Covariates for ECTLSI Teacher Impact Analysis 


Covariate 

Student level 

Female 

Black 

Hispanic 

Asian 

Other ethnicity 

Free or reduced price lunch status 

Limited English proficiency 

Special education 

Grade level indicators 

Baseline score 

Teacher level 

Black 

Hispanic 

Other ethnicity 

Years of teaching experience 

Florida Temporary Certification 

National Board Certification 

Certification area: Pre-Kindergarten/primary education 

Certification area: Preschool education 

Master's degree or higher 

School level 

Percent students with limited English proficiency 

Percent students with free or reduced price lunch status 

Percent minority students (Black and Hispanic) 

Average baseline third-grade reading score 

Average baseline third grade math score 

State accountability grade 

Block indicators 


Baseline equivalence. Exhibit A-21 presents the means and standard deviations for the baseline 
scores as well as numbers of students, virtual teachers, and schools for each of the four groups of 
teachers for math and reading analysis respectively. We further used ANOVA to test whether 
there were any statistically significant differences between any of the four groups of students. 
The p-value from the ANOVA test is shown in the last column for reading and math separately. 
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Although the ANOVA test shows a statistically significant difference in reading, the differences 
between any of the four groups are smaller than 0.25 standard deviations for reading and math 
alike. We therefore considered the four groups of students to not be substantially different at 
baseline in either reading or math, and proceeded with the analysis adjusting for the baseline 
outcome measure and covariates as described above. 


Exhibit A-21 Distributions of Baseline Test Scores for Students in Different 

Comparison Groups 



Standardized 

Mean 

Standard 

Deviation 

Student 

Sample 

N 

Virtual 

Teacher 

N 

School 

N 

P Value from 
ANOVA 

Reading 






0.002 

Nonmatched 
teachers in 
control 
schools 

0.058 

1.040 

1701 

97 

11 


Matched 
teachers in 
control 
schools 

-0.014 

0.990 

371 

27 

11 


Non-ECTLSI 
teachers in 
treatment 
schools 

-0.078 

0.981 

1378 

83 

10 


ECTLSI 

teachers 

0.036 

0.878 

408 

25 

10 


Math 






0.959 

Nonmatched 
teachers in 
control 
schools 

-0.001 

1.024 

1973 

103 

12 


Matched 
teachers in 
control 
schools 

0.017 

0.878 

378 

20 

12 


Non-ECTLSI 
teachers in 
treatment 
schools 

0.004 

1.039 

1116 

60 

8 


ECTLSI 

teachers 

-0.021 

0.868 

392 

20 

8 
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Appendix B. Implementation Fidelity 


The implementation study examined the research question: What is the fidelity of 
implementation of the Florida Master Teacher Initiative for each key program component? To 
measure participation and fidelity, SRI worked with the program developer, the University of 
Florida, to identify types and intensity of activities in each of the program components believed 
necessary to bring about the desired changes and outcomes outlined in the program logic model. 
These assumptions were used to develop the fidelity measure (Exhibit B-l). 

SRI collected administrative data each summer during the three years of the program. Fidelity 
was measured in all 20 schools implementing the intervention. To construct fidelity scores, data 
were collected on teachers’ participation in the Teacher Fellows Program; teachers’ participation 
and performance in the ECTLSI program; principals’ participation in the Principal Fellows 
program; and administrators’ and teachers’ participation in Summer Leadership Institutes. We 
used fidelity scores to assess whether each key component of the FMTI was implemented with 
fidelity for the entire sample of schools. Exhibit B-2 presents the findings on fidelity for each 
key component of the FMTI. The results for fidelity varied by program component and by year. 

The Teacher Fellow program was conducted with medium or high fidelity all three years in 
the vast majority of treatment schools. A total of 235 teachers participated in the Teacher 
Fellows program in Year 1, 242 teachers in Year 2, and 320 teachers in Year 3. This was an 
average of 30-45% of faculty at each of the schools. For teachers that chose to participate in the 
Teacher Fellows program, the majority showed active and consistent engagement. For example, 
in Year 3, 86% of Teacher Fellows attended all six sessions, 92% presented at the Learning 
Showcase, and 90% wrote and submitted a summary of their inquiry. 

The ECTLSI graduate program was implemented with medium or high fidelity in only 10 
of the 20 schools, resulting in an overall score of low fidelity for this component. Sixty-one 
teachers were enrolled in the ECTLSI program at some point. The goal had been to enroll 100 
teachers. By the end of Year 3, 29 teachers had graduated and 20 teachers were still enrolled in 
the ECTLSI program (two Cohort 1 teachers were still completing the program with Cohort 2 
teachers). The program fell short of the target of having 75% of schools meeting medium or high 
fidelity in Year 2. Since no new enrollment occurred in Year 3, the fidelity level did not increase. 
However, average fidelity scores for individual teachers in the ECTLSI program were high and 
completion rates were also good (79%). 

The Principal Fellows program only met medium or high fidelity in the first year of 
implementation. The Principal Fellows program achieved medium or high fidelity in 80% of the 
intervention schools in Year 1 but was unable to reach a sufficient level of fidelity in enough 
schools in Years 2 and 3. In Year 3, principals from only 8 of the 20 treatment schools attended 
both the statewide institute and at least one Principal Fellow meeting. Thus, in Year 3, 40% of 
schools met medium or high fidelity on the Principal Fellows Program fidelity measure. 

The Summer Leadership program achieved fidelity in only the third year of 
implementation. The fidelity of the Summer Leadership improved each year, going from 
insufficient fidelity in Years 1 and 2 to meeting fidelity benchmarks in Year 3. 

Because of low levels of implementation fidelity, the full model, as intended, could not be 
tested across all 20 schools in the initiative. However, 11 of the 20 schools had medium or 
high fidelity across all four components each of the three years. 
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Exhibit B-1: Florida Master Teacher Initiative Fidelity Measure 

Teacher Fellows Program 

Individual teacher level 


Fidelity Element 

Data Source 

Data 

Collection 

Frequency 

Scoring of 
adequacy of 
fidelity 

Attend Teacher Fellow 
meetings (up to 6). 

TF attendance roster from 
each meeting 

Annual, July 

Get a point for each 
meeting attended 

0-6 

Present project at inquiry 
showcase. 

Showcase presentation 
schedule 

Annual, July 

No = 0 

Yes = 4 

Write and submit an inquiry 
summary to be added to the 
inquiry database at UF. 

Inquiry write-up 
summaries (get from UF) 

Annual, July 

No = 0 

Yes = 5 

Teacher Fellow Fidelity 
Score 



Range 0-15 


A. Individual teacher level fidelity score for teachers participating in the Teacher Fellow 
Program 

0 points (Inadequate, if score is 0 - 7] 

1 point (Low fidelity, if score is 8 - 9] 

2 points (Medium fidelity, if score is 10-11] 

3 points (High fidelity, if score is 12 and more] 

B. School level: 

Bla. Percent of teachers in a school who participate (enroll) in the Teacher Fellows 
program 

0 points (Inadequate, if 0% - 5% participate] 

1 point (Low fidelity, if 6%-10% participate] 

2 points (Medium fidelity, if ll%-25% participate] 

3 points (High fidelity, if 26% or more participate] 

Bib. Number of Teacher Fellow meetings held 

0 points (Inadequate, if 0-1 meetings] 

1 point (Low fidelity, if 2-3 meetings] 

2 points (Medium fidelity, if 4-5 meetings] 

3 points (High fidelity, if 6 meetings] 

B2. Average of individual teacher fidelity score for teachers participating in the 
Teacher Fellow program 

Calculate average of individual teacher points for school level score (A], 

Overall School Level = fBla + Bib ) + B2 (range 0-6 points) 

2 

C. Overall Project Level: Adequate Fidelity = At least 75% of schools have a score of 4 
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Exhibit B-1 (continued): Florida Master Teacher Initiative Fidelity Measure 

Teachers in Early Childhood Graduate Degree Program (ECTLSI) 

ANNUAL SCORE 


Fidelity Element 

Data Source 

Data 

Collection 

Frequency 

Scoring of adequacy of 
fidelity 

Maintains 3.0 GPA 

UF administrative 
data 

Annual, July 

No = 0 

Yes = 5 

Completes scheduled 
classes for the year 

Year 1:16 credits 

Year 2:17 credits 

Year 3: 6 credits 

UF administrative 
data 

Annual, July 

Completed all credits 

No = 0 

Yes = 10 

Attends facilitator training 
(ever) 

Attendance forms 

Annual, July 

Gets a point for each 
facilitator training 
attended (up to 2) 

0-2 (cumulative) 

Facilitates Teacher Fellow 
or other inquiry group, 
leads professional 
development, or facilitates 
PLC (ever) 

Teacher survey 

Annual, July 

Gets a point for each 
group led (up to 2) 

0-2 (cumulative) 

Completes an inquiry 
project during program 
(ever) 

Teacher survey 

Annual, July 

Gets a point for each 
inquiry project 
completed (up to 2) 0-2 
(cumulative) 

ECTLSI Fidelity Score 



Range 0-21 


A. Individual teacher level 

0 points (Inadequate, if score is 9 or lower) 

1 point (Low fidelity, if score is 10-14) 

2 points (Medium fidelity, if score is 15-16) 

3 points (High fidelity, if score is 17 or higher) 

B. School level: 

Bl. Number of teachers in a school who start a first class in the ECTLSI program 
(2011-2015) 

0 points (Inadequate, if 0 participate) 

1 point (Low fidelity, if 1 participates) 

2 points (Medium fidelity, if 2-3 participate) 

3 points (High fidelity, if 4 or more participate) 

B2. Average of individual teacher fidelity scores of teachers in ECTLSI (2011-2015) 

Calculate average of individual teacher points for school level score. 

Overall School Level = Bl + B2 (range 0-6 points) 

C. Overall project level: Adequate Fidelity = At least 75% of schools have a score of 5 
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Exhibit B-1 (continued): Florida Master Teacher Initiative Fidelity Measure 

Principal Fellows 


Fidelity Element 

Data Source 

Data Collection 
Frequency 

Scoring of adequacy 
of fidelity 

Attend Principal 

Fellow meetings. 

(up to 4 points) 

Principal Fellow 
attendance roster, for 
each meeting 

Annual, July 

Get a point for each 
meeting attended 

0-4 (no point for a 
fifth meeting in 

Year 1) 

Attend principal 
institute (s) 

Year 1: 2 meetings 

Year 2: 1 meeting 

Year 3: 1 meeting 

Principal Fellow 
attendance roster, for 
each meeting 

Annual, July 

Get 4 points attending 
a least one institute a 
year 

Principal Fellow 
Fidelity Score 



Range 0-8 


B. School level 

0 points (Inadequate, if score is 0-2) 

1 point (Low fidelity, if score is 3-4) 

2 points (Medium fidelity, if score is 5-6) 

3 points (High fidelity, if score is 7 or more) 

C. Overall project level: Adequate Fidelity = At least 75% of schools have a score of 2 
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Exhibit B-1 (continued): Florida Master Teacher Initiative Fidelity Measure 

Summer Leadership Institutes 


Fidelity Element 

Data Source 

Data Collection 
Frequency 

Scoring of adequacy of 
fidelity 

Principal attends 

Sign-in/admin, data 

Annual, July 

No = 0 

Yes = 3 

Asst, principal attends 

Sign-in/admin, data 

Annual, July 

No = 0 

Yes = 1 

Teacher leader(s) or 
community 
involvement specialist 

Sign-in/admin, data 

Annual, July 

No = 0 

Yes = 1 point/attendee 
0-4 

A teacher in TLSI 
graduate program 

Sign-in/admin, data 

Annual, July 

No = 0 

Yes = 1 

Develop a school 
action plan 

UF administrative data 

Annual, July 

No = 0 

Yes = 3 

Summer Leadership 
Institute Fidelity 

Score 



Range 0-12 


B. School level 

0 points (Inadequate, if score is 0-3) 

1 point (Low fidelity, if score is 4-6) 

2 points (Medium fidelity, if score is 7-9) 

3 points (High fidelity, if score is 10 or more) 

C. Overall project level: Adequate Fidelity = At least 75% of schools have a score of 2 
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Exhibit B-2. Implementation with Fidelity for each Key Component of the Intervention, at the Sample Level 


Key 

Components 
(from Logic 
Model) 

Measurement 
(see Fidelity 
Measure 
attachment) 

Definition of 
implementation 
“with fidelity” at 
sample level 

Year 1 

% of 

schools 

meeting 

fidelity 

threshold 

for 

component 

Year 1 

Was the 
component 
implemented 
with fidelity 
at the 
sample 
level? (Y/N) 

Year 2 

% of 

schools 

meeting 

fidelity 

threshold 

for 

component 

Year 2 

Was the 
component 
implemented 
with fidelity 
at the 
sample 
level? (Y/N) 

Year 3 

% of 

schools 

meeting 

fidelity 

threshold 

for 

component 

Year 3 

Was the 
component 
implemented 
with fidelity 
at the 
sample 
level? (Y/N) 

Teacher 

Fellows 

Program 

School-level 

score, 

0-6 point 
scale 

At least 75% of 
schools have a 
score of 4 or 
higher 

95% 

YES 

90% 

YES 

100% 

YES 

Early 

Childhood 

Education 

Graduate 

Program 

School-level 

score, 

0-6 point 
scale 

At least 75% of 
schools have a 
score of 5 or 
higher 

25% 

NO 

50% 

NO 

50% 

NO 

Principal 

Fellows 

Program 

School-level 

score, 

0-3 point 
scale 

At least 75% of 
schools have a 
score of 2 or 
higher 

80% 

YES 

55% 

NO 

40% 

NO 

Summer 

Leadership 

Institutes 

School-level 

score, 

0-3 point 
scale 

At least 75% of 
schools have a 
score of 2 or 
higher 

55% 

NO 

70% 

NO 

90% 

YES 
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Appendix C. Impact Estimates 


Table C-1. Confirmatory Analysis of Schoolwide Impact on Teacher Survey 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Learner-centered instruction 

-0.073 

0.049 

0.142 

1190 

40 

Assessment-informed practice 

-0.054 

0.044 

0.217 

1184 

40 

Developmentally appropriate practices 

-0.033 

0.047 

0.473 

1188 

40 

Emphasis on higher order thinking skills 

0.002 

0.036 

0.962 

1194 

40 

Differentiated instruction 

-0.080 

0.035 

0.024 

1194 

40 

Culturally responsive instruction 

-0.044 

0.061 

0.469 

1184 

40 

Collaboration around instruction 

-0.015 

0.079 

0.845 

1189 

40 

Trusting relationships between teachers 

0.015 

0.056 

0.790 

1186 

40 

Principal leadership 

0.065 

0.080 

0.411 

1178 

40 

Teacher leadership 

0.050 

0.071 

0.477 

1188 

40 

Family partnerships 

0.061 

0.062 

0.326 

1189 

40 

Regular use of variety of assessments 

-0.042 

0.046 

0.363 

1186 

40 

Early childhood instructional knowledge 

0.035 

0.039 

0.363 

1180 

40 

General instructional knowledge 

0.024 

0.034 

0.481 

1180 

40 

Governance activities 

0.380 

0.189 

0.044 

1186 

40 

Early learning outreach activities 

0.356 

0.182 

0.050 

1186 

40 
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Table C-2. Exploratory Analysis of Schoolwide Impact on Teacher Survey in 

Medium- to High-fidelity Schools 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Learner-centered instruction 

0.048 

0.110 

0.664 

968 

29 

Assessment-informed practice 

0.042 

0.094 

0.654 

964 

29 

Developmentally appropriate practices 

0.087 

0.103 

0.397 

966 

29 

Emphasis on higher order thinking skills 

0.078 

0.085 

0.355 

971 

29 

Differentiated instruction 

-0.021 

0.079 

0.791 

971 

29 

Culturally responsive instruction 

0.126 

0.141 

0.373 

964 

29 

Collaboration around instruction 

0.219 

0.159 

0.167 

968 

29 

Trusting relationships between teachers 

0.216 

0.107 

0.043 

965 

29 

Principal leadership 

0.213 

0.148 

0.151 

963 

29 

Teacher leadership 

0.359 

0.145 

0.013 

968 

29 

Family partnerships 

0.342 

0.124 

0.006 

968 

29 

Regular use of variety of assessments 

0.233 

0.100 

0.020 

966 

29 

Early childhood instructional knowledge 

0.085 

0.083 

0.308 

961 

29 

General instructional knowledge 

0.094 

0.080 

0.237 

961 

29 

Governance activities 

-0.281 

0.412 

0.495 

967 

29 

Early learning outreach activities 

0.556 

0.395 

0.160 

967 

29 


Table C-3. Confirmatory Analysis of Schoolwide Impact on Students 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Math achievement 

-0.037 

0.037 

0.325 

10587 

40 

Reading achievement 

-0.037 

0.020 

0.068 

10610 

40 


Table C-4. Exploratory Analysis of Schoolwide Impact on Students Who 

Remained in the Same School 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Math achievement, grades 1 & 2 at baseline 

-0.023 

0.058 

0.687 

4732 

40 

Reading achievement, grades 1 & 2 at baseline 

-0.021 

0.028 

0.450 

4740 

40 

Math achievement, grades PK & K at baseline 

0.012 

0.066 

0.854 

2571 

40 

Reading achievement, grades PK & K at baseline 

-0.005 

0.038 

0.888 

2577 

40 
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Table C-5. Exploratory Analysis of Schoolwide Impact on Students Who 
Remained in Medium- to High-fidelity FMTI Schools 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Math achievement, grades 1 & 2 at baseline 

-0.076 

0.091 

0.403 

3859 

29 

Reading achievement, grades 1 & 2 at baseline 

-0.075 

0.054 

0.166 

3867 

29 

Math achievement, grades PK & K at baseline 

0.020 

0.121 

0.870 

2062 

29 

Reading achievement, grades PK & K at baseline 

-0.106 

0.092 

0.248 

2067 

29 


Table C-6. Confirmatory Analysis of ECTLSI Impact on Teacher Survey Outcomes 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

Learner-centered instruction 

-0.098 

0.084 

0.246 

439 

33 

Assessment-informed practice 

0.048 

0.126 

0.704 

440 

33 

Developmentally appropriate practices 

-0.068 

0.137 

0.619 

440 

33 

Emphasis on higher order thinking skills 

-0.004 

0.075 

0.957 

445 

33 

Differentiated instruction 

0.054 

0.110 

0.622 

444 

33 

Culturally responsive instruction 

0.118 

0.164 

0.472 

436 

33 

Collaboration around instruction 

0.238 

0.145 

0.101 

443 

32 

Trusting relationships between teachers 

0.119 

0.132 

0.369 

441 

32 

Principal leadership 

-0.158 

0.086 

0.064 

442 

32 

Teacher leadership 

0.444 

0.189 

0.019 

445 

33 

Family partnerships 

-0.303 

0.187 

0.105 

442 

32 

Regular use of variety of assessments 

-0.155 

0.159 

0.330 

440 

33 

Early childhood instructional knowledge 

0.380 

0.101 

0.000 

443 

33 

General instructional knowledge 

0.273 

0.100 

0.006 

443 

33 

Governance activities 

2.713 

1.198 

0.023 

446 

33 

Early learning outreach activities 

1.234 

0.535 

0.021 

446 

33 
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Table C-7. Confirmatory Analysis of ECTLSI Impact on CLASS Outcomes 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Teacher 

N 

School 

N 

CLASS emotional support score 

-0.035 

0.306 

0.910 

62 

26 

CLASS classroom organization score 

-0.391 

0.366 

0.285 

62 

26 

CLASS instructional support score 

1.657 

0.434 

0.000 

62 

26 


Table C-8. Confirmatory Analysis of ECTLSI Impact on Student Outcomes 



Impact 

Estimate 

Impact 

Estimate 

Standard 

Error 

Impact 

Estimate 

p-Value 

Student 

N 

Teacher 

N 

School 

N 

Math achievement 

-0.201 

0.150 

0.179 

3859 

203 

20 

Reading achievement 

0.158 

0.109 

0.147 

3858 

232 

21 
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